PhpDig.net - View Single Post

Psycho1 · 04-17-2004, 09:52 AM

I just installed PHPDig on my site today and I'm impressed! It followed my robots.txt file so it didn't crawl my forums.....but I have a mod for phpBB that adds posts from a forum on my homepage....and it has several pages of posts on my homepage with urls like mysite.com/index.php?start=5&sid=afdhjdsahfksdahkflafsadkhjfasfs

It likes to visit these types of links many, many times before it moves on to other parts of my site. Is it possible for PHPDig to be setup to only crawl php pages that do not have variables in the url? And how would I go about stopping the spidering so I can fix this issue or can I fix this without stopping the spidering process?

Thanks in advance for any help.

04-17-2004, 09:52 AM	#1
Psycho1 Green Mole Join Date: Apr 2004 Posts: 9	PHP spidering nuisance I just installed PHPDig on my site today and I'm impressed! It followed my robots.txt file so it didn't crawl my forums.....but I have a mod for phpBB that adds posts from a forum on my homepage....and it has several pages of posts on my homepage with urls like mysite.com/index.php?start=5&sid=afdhjdsahfksdahkflafsadkhjfasfs It likes to visit these types of links many, many times before it moves on to other parts of my site. Is it possible for PHPDig to be setup to only crawl php pages that do not have variables in the url? And how would I go about stopping the spidering so I can fix this issue or can I fix this without stopping the spidering process? Thanks in advance for any help.