PhpDig.net - View Single Post

davenewt · 07-12-2004, 06:56 AM

Server logs... thought that's what you meant... anyway, seems I had to "Dig This" to add the site to start with, then use the "Update Form" part of the Admin interface and click the green tick icon next to "Root" to re-index.

However...

Doing this initially started to pick up all links to the forum directory, so I added a robots.txt file to exclude this directory.

Now I have discovered the main problem. The only navigation on the site so far is via a dynamic javascript menu which is added to the HTML at runtime. There are no links (except to the aforementioned forums) in the body of the HTML.

How can I spider sub-directories which aren't linked to from the root index file? Do I need to add a line of navigation to the bottom of the index page which will make the spider aware of the subdirectories, or can I tell the spider to look for them some other way?

Almost there

Thanks so much for the help.

Cheers,
Dave.

07-12-2004, 06:56 AM	#18
davenewt Green Mole Join Date: Jul 2004 Posts: 20	Server logs... thought that's what you meant... anyway, seems I had to "Dig This" to add the site to start with, then use the "Update Form" part of the Admin interface and click the green tick icon next to "Root" to re-index. However... Doing this initially started to pick up all links to the forum directory, so I added a robots.txt file to exclude this directory. Now I have discovered the main problem. The only navigation on the site so far is via a dynamic javascript menu which is added to the HTML at runtime. There are no links (except to the aforementioned forums) in the body of the HTML. How can I spider sub-directories which aren't linked to from the root index file? Do I need to add a line of navigation to the bottom of the index page which will make the spider aware of the subdirectories, or can I tell the spider to look for them some other way? Almost there Thanks so much for the help. Cheers, Dave.