Command Line Spider spiders all sites

Wayne McBryde · 01-27-2004, 12:50 PM

I’m still working to install 1.8.0. I’m building a new database and have a LOT of sites to spider. I created 9 text files with domain names, url_list_1.txt through url_list_9.txt.
When I entered “php –f spider.php url_list_1.txt” the spider, spidered the sites in the text file. When I enter “php –f spider.php url_list_2.txt” the spider, spiders the sites in list 2 then respiders the sites from list 1. Is this normal, or am I doing something wrong?

Charter · 01-27-2004, 01:57 PM

Hi. If you are still using version 1.6.5, then PhpDig will spider similar to that. Once you upgrade to 1.8.0, only the ULRs in each file will be crawled.

Wayne McBryde · 01-27-2004, 04:37 PM

It is 1.8.0 that I am having this problem with.

Charter · 01-27-2004, 05:15 PM

Hi. Between runs, check that the tempspider table is empty. If it's not empty, then empty it. You can do this by clicking the delete button from the admin panel without selecting a site, or run the following query:

Code:

DELETE FROM tempspider;

Sometimes things can get left in the tempspider table when there is no error but the corresponding page hasn't been indexed. This can happen if the spidering process is terminated prematurely.

01-27-2004, 12:50 PM	#1
Wayne McBryde Orange Mole Join Date: Oct 2003 Location: NC, USA Posts: 34	Command Line Spider spiders all sites I’m still working to install 1.8.0. I’m building a new database and have a LOT of sites to spider. I created 9 text files with domain names, url_list_1.txt through url_list_9.txt. When I entered “php –f spider.php url_list_1.txt” the spider, spidered the sites in the text file. When I enter “php –f spider.php url_list_2.txt” the spider, spiders the sites in list 2 then respiders the sites from list 1. Is this normal, or am I doing something wrong? __________________ Wayne Mcbryde http://LakeNormansWeb.com We search all of Lake Norman!

01-27-2004, 01:57 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. If you are still using version 1.6.5, then PhpDig will spider similar to that. Once you upgrade to 1.8.0, only the ULRs in each file will be crawled. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

01-27-2004, 04:37 PM	#3
Wayne McBryde Orange Mole Join Date: Oct 2003 Location: NC, USA Posts: 34	It is 1.8.0 that I am having this problem with. __________________ Wayne Mcbryde http://LakeNormansWeb.com We search all of Lake Norman!

01-27-2004, 05:15 PM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Between runs, check that the tempspider table is empty. If it's not empty, then empty it. You can do this by clicking the delete button from the admin panel without selecting a site, or run the following query: Code: DELETE FROM tempspider; Sometimes things can get left in the tempspider table when there is no error but the corresponding page hasn't been indexed. This can happen if the spidering process is terminated prematurely. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
spider from command line	twanoo	Troubleshooting	3	01-14-2005 10:04 AM
How To call spider from command line with debth options?	jburnett	How-to Forum	1	01-12-2005 01:03 PM
Spider.php is killed at the command line	bforsyth	Troubleshooting	2	12-05-2004 01:09 PM
Problem running spider from Command Line	joshuag200	Troubleshooting	17	09-13-2004 07:57 PM
Spider in command line : 3 errors	Yannick	Troubleshooting	2	12-19-2003 03:01 AM