Spider not finding anything.

nvahalik · 01-25-2005, 11:05 AM

I'm trying to spider a site I'm typing in the site http://www.mysite.com/subdir/

I want it to go through everything, not just subdir/ but it never finds any links follow. Is there a debug mode to watch what the spider is doing?

I have also edited the config file:

PHP Code:


			
define('LIMIT_TO_DIRECTORY',false);

nvahalik · 01-25-2005, 11:37 AM

Output of spider.php (using web interface):
Spidering in progress... [Stop spider]
SITE : http://www2.mysite.com
Exclude paths :
- @NONE@
1:http://www2.mysite.com/
(time : 00:00:05)
No link in temporary table
links found : 1
http://www2.mysite.com/
Optimizing tables...
Indexing complete ! [Back] to admin interface.

Dave A · 01-25-2005, 02:41 PM

I have seen these type of results before if the depth and number of links has not been set before you index a web site, perhaps you should check the settings in the admin panel for what depth and number of links to be indexed.
One other thing to try is to use the update site feature which sometimes re indexes better the second time around, it will just mark a red cross by the page if it has already been indexed.
When I fisrt index a web site I go in quite shallow and follow it up a while later with amuch deper index.
Give it another try it should be okay

01-25-2005, 11:05 AM	#1
nvahalik Green Mole Join Date: Jan 2005 Posts: 2	Spider not finding anything. I'm trying to spider a site I'm typing in the site http://www.mysite.com/subdir/ I want it to go through everything, not just subdir/ but it never finds any links follow. Is there a debug mode to watch what the spider is doing? I have also edited the config file: PHP Code: `define('LIMIT_TO_DIRECTORY',false);` Last edited by nvahalik; 01-25-2005 at 11:13 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Grep command for finding Saga zero length	Dave A	Troubleshooting	2	02-06-2006 03:00 PM
Using phpdig for finding copyright infringements	leto	How-to Forum	0	09-22-2005 01:31 AM
finding dead links	manute	How-to Forum	2	01-14-2004 04:00 PM

01-25-2005, 11:37 AM	#2
nvahalik Green Mole Join Date: Jan 2005 Posts: 2	Output of spider.php (using web interface): Spidering in progress... [Stop spider] SITE : http://www2.mysite.com Exclude paths : - @NONE@ 1:http://www2.mysite.com/ (time : 00:00:05) No link in temporary table links found : 1 http://www2.mysite.com/ Optimizing tables... Indexing complete ! [Back] to admin interface.

01-25-2005, 02:41 PM	#3
Dave A Purple Mole Join Date: Aug 2004 Location: North Island New Zealand Posts: 170	I have seen these type of results before if the depth and number of links has not been set before you index a web site, perhaps you should check the settings in the admin panel for what depth and number of links to be indexed. One other thing to try is to use the update site feature which sometimes re indexes better the second time around, it will just mark a red cross by the page if it has already been indexed. When I fisrt index a web site I go in quite shallow and follow it up a while later with amuch deper index. Give it another try it should be okay