Spidering not completing

lentilsoup · 11-20-2006, 03:18 AM

I used PHPDig on my local machine and successfully spidered and searched my website. I then uploaded this code to a webserver and it stopped working properly.

At first it would not find anything. On 'digging' a new site it would say it had completed indexing very quickly and then on return to the admin page it would say 1 host and zero for everything else (keywords etc..) in the statistics. After fiddling around I eventually indexed some pages, but not all. I'm sure I didn't really change anything. It seems a little sporadic.

So the problem is: the spider doesn't seem to be following all links as it has only found around 9 pages whereas the site has around 100 pages. I change nothing since last night and now it won't find any links again.

Here's a checklist of things I have done based on previous posts and/or the documentation:

- 'safe-mode' is off: Check
- 'allow_url_fopen' is on: Check
- Site is linked up entirely by simple <a href> tags: Check
- Tried the DEFINES as follows:
define('ABSOLUTE_SCRIPT_PATH',correct with no trailing slash);
define('LIMIT_TO_DIRECTORY',false);
define('PHPDIG_SESSID_REMOVE',true);
define('PHPDIG_IN_DOMAIN',true);
Check
- The three directories that require it are CHMOD 777: Check
- Tried the $pu['port'] = ''; bugfix - both as '' and '0': Check

Could this be something to do with the commercial server I am running this on? Or is it more likely the code?

Dave A · 11-20-2006, 04:25 AM

This could be a problem with the way the new sever balances it loading, some odd things have turned up depending on how the laod balancing is set up

lentilsoup · 11-20-2006, 10:24 AM

Is there any way around this if I have no control over the server myself?

11-20-2006, 03:18 AM	#1
lentilsoup Green Mole Join Date: Nov 2006 Posts: 2	Spidering not completing I used PHPDig on my local machine and successfully spidered and searched my website. I then uploaded this code to a webserver and it stopped working properly. At first it would not find anything. On 'digging' a new site it would say it had completed indexing very quickly and then on return to the admin page it would say 1 host and zero for everything else (keywords etc..) in the statistics. After fiddling around I eventually indexed some pages, but not all. I'm sure I didn't really change anything. It seems a little sporadic. So the problem is: the spider doesn't seem to be following all links as it has only found around 9 pages whereas the site has around 100 pages. I change nothing since last night and now it won't find any links again. Here's a checklist of things I have done based on previous posts and/or the documentation: - 'safe-mode' is off: Check - 'allow_url_fopen' is on: Check - Site is linked up entirely by simple <a href> tags: Check - Tried the DEFINES as follows: define('ABSOLUTE_SCRIPT_PATH',correct with no trailing slash); define('LIMIT_TO_DIRECTORY',false); define('PHPDIG_SESSID_REMOVE',true); define('PHPDIG_IN_DOMAIN',true); Check - The three directories that require it are CHMOD 777: Check - Tried the $pu['port'] = ''; bugfix - both as '' and '0': Check Could this be something to do with the commercial server I am running this on? Or is it more likely the code?

11-20-2006, 04:25 AM	#2
Dave A Purple Mole Join Date: Aug 2004 Location: North Island New Zealand Posts: 170	This could be a problem with the way the new sever balances it loading, some odd things have turned up depending on how the laod balancing is set up

11-20-2006, 10:24 AM	#3
lentilsoup Green Mole Join Date: Nov 2006 Posts: 2	Is there any way around this if I have no control over the server myself?