PhpDig.net - View Single Post - Keeping the spider in the search directory and its subdirs

ciaran@clissman · 05-24-2004, 05:59 AM

Hi,
I typically want to spider just a subdirectory and its subdirs, so that I don't want the spider to go up into the parent directory of the URL that I specify.

e.g. I want to index all of www.myplace.com/searchme

The starting point is www.myplace.com/searchme/index.html
I want all the other stuff in /searchme to be indexed, but I don't want www.myplace.come/donttouch, EVEN THOUGH there is a link from /searchme/index.html to /donttouch/index.html.

IS there a way to tell PHPdig not to 'go up' in the directory hierarchy ?

Thanks a lot !

05-24-2004, 05:59 AM	#1
ciaran@clissman Green Mole Join Date: May 2004 Posts: 10	Keeping the spider in the search directory and its subdirs Hi, I typically want to spider just a subdirectory and its subdirs, so that I don't want the spider to go up into the parent directory of the URL that I specify. e.g. I want to index all of www.myplace.com/searchme The starting point is www.myplace.com/searchme/index.html I want all the other stuff in /searchme to be indexed, but I don't want www.myplace.come/donttouch, EVEN THOUGH there is a link from /searchme/index.html to /donttouch/index.html. IS there a way to tell PHPdig not to 'go up' in the directory hierarchy ? Thanks a lot !