I'm having the same problem indexing my site
http://www.kevinfreitas.net/ and do have allow_url_fopen enabled. The install looks like it created everything in the DB just fine and I believe I have the correct perms for all the files (although I don't know this for sure since the oft posted link
http://www.phpdig.net/navigation.php?action=doc doesn't seem to work).
Any ideas?
Thanks and take care! ~ WunderStump