Command line spider ignoring "filelist.txt"

lighthouse · 08-03-2004, 07:12 AM

I can run a command line spider by doing the following:

>> php - f search.php
The above indexes everything...

However - I want to just update a few sites, so I have a file called "filelist.txt" which contains 2 URLS's (each on a diffent line)...

When I do the following:

>>php - f search.php filelist.txt
It makes no difference... phpdig proceeds to update everything!!!!

Any ideaS?...
I need to resolve this so i can setup a cron....

Also - FYI:
>> php -f search.php http://www.anydomain.com
This works as expected... It's just with specifying a filelist that there is problems.

Thanks in advance
LH

Charter · 08-03-2004, 12:29 PM

Hi. What version of PhpDig are you using?

lighthouse · 08-09-2004, 04:31 AM

1.6.x (as far as i can see - taken from "spider.php" file)

lighthouse · 08-09-2004, 06:08 AM

Quote:

Originally posted by lighthouse
1.6.x (as far as i can see - taken from "spider.php" file)

Update - I upgraded to latest version, and STILL it ignores the filelist.txt and simply begins indexing everything!!!

Please help!!
LH

lighthouse · 08-10-2004, 06:09 AM

Update:
Have tried and tried with this, but no luck.

I've opted to just run 6 crons (instead of 1), in the format:
php - f path/to/dir/spider.php http://theurliwanttoindex.com

Not pretty, but it works....
Still open to any suggestions if you have any...

LH

Charter · 08-15-2004, 04:03 PM

Hi. Try updating to 1.8.3.

lighthouse · 08-16-2004, 06:32 AM

Quote:

Originally posted by Charter
Hi. Try updating to 1.8.3.

Charter - I did upgrade - and still the same result....

Charter · 08-17-2004, 03:27 PM

Is the tempspider table empty?

lighthouse · 08-18-2004, 02:01 AM

That seems to have done the trick!!!!

Does the Temporary table always need to be empty then?...
Only I see that it's often got entries in it...

LH

Charter · 08-18-2004, 08:35 AM

Hi. The tempspider table should try and empty after PhpDig is done, but if it doesn't (for example, the spidering process prematurely ends) then just go and empty that table.

08-03-2004, 07:12 AM	#1
lighthouse Green Mole Join Date: Mar 2004 Posts: 20	Command line spider ignoring "filelist.txt" I can run a command line spider by doing the following: >> php - f search.php The above indexes everything... However - I want to just update a few sites, so I have a file called "filelist.txt" which contains 2 URLS's (each on a diffent line)... When I do the following: >>php - f search.php filelist.txt It makes no difference... phpdig proceeds to update everything!!!! Any ideaS?... I need to resolve this so i can setup a cron.... Also - FYI: >> php -f search.php http://www.anydomain.com This works as expected... It's just with specifying a filelist that there is problems. Thanks in advance LH

08-03-2004, 12:29 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. What version of PhpDig are you using? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

08-15-2004, 04:03 PM	#6
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Try updating to 1.8.3. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

08-17-2004, 03:27 PM	#8
Charter Head Mole Join Date: May 2003 Posts: 2,539	Is the tempspider table empty? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

08-18-2004, 08:35 AM	#10
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. The tempspider table should try and empty after PhpDig is done, but if it doesn't (for example, the spidering process prematurely ends) then just go and empty that table. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

08-09-2004, 04:31 AM	#3
lighthouse Green Mole Join Date: Mar 2004 Posts: 20	1.6.x (as far as i can see - taken from "spider.php" file)

08-10-2004, 06:09 AM	#5
lighthouse Green Mole Join Date: Mar 2004 Posts: 20	Update: Have tried and tried with this, but no luck. I've opted to just run 6 crons (instead of 1), in the format: php - f path/to/dir/spider.php http://theurliwanttoindex.com Not pretty, but it works.... Still open to any suggestions if you have any... LH

08-18-2004, 02:01 AM	#9
lighthouse Green Mole Join Date: Mar 2004 Posts: 20	That seems to have done the trick!!!! Does the Temporary table always need to be empty then?... Only I see that it's often got entries in it... LH

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
shows blank page if "Search All" and "exact phrase", timeout?	alokjain9	Troubleshooting	2	03-07-2006 08:08 AM
"search depth" and "links per" features	laurentxav	How-to Forum	1	01-12-2005 08:27 PM
"from" line in email	hollycrj	Coding & Tutorials	2	06-29-2004 01:34 PM
PhpDig Ignoring Something in robots.txt	Destroyer X	Troubleshooting	2	06-18-2004 02:57 PM