![]() |
|
![]() |
#1 |
Green Mole
Join Date: Jan 2005
Posts: 9
|
Will not follow links
I feel really stupid that I can install this but I can not seem to make it work. I have just installed 1.8.7. Connect to DB fine. When I enter a URL to spider, it only crawls one page. I have tried a couple of URLs and get the same results. When entering the URL I set depth to 20 and links to 20 as I want it to crawl as deep as possible. I also have tried it from the command line and get the same results:
[prompt]# php -f ./spider.php http://www.fake_domain.com/destinations/ 6485: old priority 0, new priority 18 Spidering in progress... ----------------------------- SITE : http://www.fake_domain.com/ Exclude paths : - @NONE@ XDuplicate of an existing document 1:http://www.fake_domain.com/destinations/ (time : 00:00:09) No link in temporary table links found : 1 Optimizing tables... Indexing complete ! Can someone please tell me what I am doing wrong. Thanks T |
![]() |
![]() |
![]() |
#2 |
Green Mole
Join Date: Sep 2004
Posts: 19
|
|
![]() |
![]() |
![]() |
#3 |
Green Mole
Join Date: May 2004
Posts: 25
|
I've been running phpdig for a while on on our site, and despite some command-line issues, it's been working just fine in spidering our site. But when I try to get it to spider a second site, it comes up with nothing (as described above). Any idea why the spider would work for one site and not another?
|
![]() |
![]() |
![]() |
|
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
spider.php will NOT follow link | barik | Troubleshooting | 0 | 10-23-2004 06:07 PM |
it founds all the link but does not follow them all | Fking | Troubleshooting | 2 | 10-06-2004 08:19 AM |
Making Search Spider Follow Links? | demo | Coding & Tutorials | 1 | 08-08-2004 05:22 PM |
don't follow link | Onno | How-to Forum | 1 | 03-05-2004 09:45 AM |
Relative links to follow in meta-refresh and Javascript | obottek | Mod Requests | 4 | 10-05-2003 10:20 AM |