Will not follow links

td234 · 01-17-2005, 02:44 PM

I feel really stupid that I can install this but I can not seem to make it work. I have just installed 1.8.7. Connect to DB fine. When I enter a URL to spider, it only crawls one page. I have tried a couple of URLs and get the same results. When entering the URL I set depth to 20 and links to 20 as I want it to crawl as deep as possible. I also have tried it from the command line and get the same results:

[prompt]# php -f ./spider.php http://www.fake_domain.com/destinations/
6485: old priority 0, new priority 18
Spidering in progress...
-----------------------------
SITE : http://www.fake_domain.com/
Exclude paths :
- @NONE@
XDuplicate of an existing document
1:http://www.fake_domain.com/destinations/
(time : 00:00:09)

No link in temporary table
links found : 1
Optimizing tables...
Indexing complete !

Can someone please tell me what I am doing wrong.

Thanks

T

Warmbells · 01-17-2005, 02:57 PM

Try here

http://www.phpdig.net/forum/showthre...ht=links+found

drywall · 01-27-2005, 05:37 AM

I've been running phpdig for a while on on our site, and despite some command-line issues, it's been working just fine in spidering our site. But when I try to get it to spider a second site, it comes up with nothing (as described above). Any idea why the spider would work for one site and not another?

01-17-2005, 02:44 PM	#1
td234 Green Mole Join Date: Jan 2005 Posts: 9	Will not follow links I feel really stupid that I can install this but I can not seem to make it work. I have just installed 1.8.7. Connect to DB fine. When I enter a URL to spider, it only crawls one page. I have tried a couple of URLs and get the same results. When entering the URL I set depth to 20 and links to 20 as I want it to crawl as deep as possible. I also have tried it from the command line and get the same results: [prompt]# php -f ./spider.php http://www.fake_domain.com/destinations/ 6485: old priority 0, new priority 18 Spidering in progress... ----------------------------- SITE : http://www.fake_domain.com/ Exclude paths : - @NONE@ XDuplicate of an existing document 1:http://www.fake_domain.com/destinations/ (time : 00:00:09) No link in temporary table links found : 1 Optimizing tables... Indexing complete ! Can someone please tell me what I am doing wrong. Thanks T

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
spider.php will NOT follow link	barik	Troubleshooting	0	10-23-2004 06:07 PM
it founds all the link but does not follow them all	Fking	Troubleshooting	2	10-06-2004 08:19 AM
Making Search Spider Follow Links?	demo	Coding & Tutorials	1	08-08-2004 05:22 PM
don't follow link	Onno	How-to Forum	1	03-05-2004 09:45 AM
Relative links to follow in meta-refresh and Javascript	obottek	Mod Requests	4	10-05-2003 10:20 AM

01-17-2005, 02:57 PM	#2
Warmbells Green Mole Join Date: Sep 2004 Posts: 19	Try here http://www.phpdig.net/forum/showthre...ht=links+found

01-27-2005, 05:37 AM	#3
drywall Green Mole Join Date: May 2004 Posts: 25	I've been running phpdig for a while on on our site, and despite some command-line issues, it's been working just fine in spidering our site. But when I try to get it to spider a second site, it comes up with nothing (as described above). Any idea why the spider would work for one site and not another?