Error while spidering

jimigisme · 10-01-2003, 01:50 PM

I've got phpdig running from the command line as follows:
C:\php>php.exe -f C:\phpdig\admin\spider.php sites.txt
THe spider runs for a bit:
Spidering in progress...
Then:
Notice: Undefined index: path in C:\phpdig\admin\robot_functions.php on line 347
Any ideas would be helpful,
Thank you!

Charter · 10-01-2003, 07:26 PM

What's in sites.txt, i.e., is there a full URL, one per line?

jimigisme · 10-02-2003, 09:27 AM

Example of sites.txt
http://www.sitename.com
http://www.sitename.com
http://www.sitename.com
http://www.sitename.com

Charter · 10-05-2003, 10:37 AM

Where is sites.txt located? Line 347 in robot_functions.php is:

PHP Code:


			
$path = $newurl["path"];

Could you change that to:

PHP Code:


			
$path = $newurl["path"];

echo $path . " <--- Path<br><br>";

to see what $path is?

jimigisme · 10-06-2003, 09:03 AM

Thank you for your help, here is what I got:

SITE : http://www.welcometoourworldofgifts.com/
Exclude paths :
- @NONE@
/welcome/default.asp <--- Path<br><br>+1:http://www.welcom
(time : 00:00:28)
/welcome/default.asp <--- Path<br><br>/welcome/default.asp

Charter · 10-06-2003, 03:51 PM

Hmm... How about try the following:

In the config file, set the following:

PHP Code:


			
define('PHPDIG_SESSID_REMOVE',true); // remove SIDS from indexed URLS

define('PHPDIG_SESSID_VAR','mscssid'); // name of the SID variable

Anyone out there ever have problems indexing *.asp files?

10-01-2003, 01:50 PM	#1
jimigisme Green Mole Join Date: Sep 2003 Posts: 10	Error while spidering I've got phpdig running from the command line as follows: C:\php>php.exe -f C:\phpdig\admin\spider.php sites.txt THe spider runs for a bit: Spidering in progress... Then: Notice: Undefined index: path in C:\phpdig\admin\robot_functions.php on line 347 Any ideas would be helpful, Thank you!

10-01-2003, 07:26 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	What's in sites.txt, i.e., is there a full URL, one per line? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

10-05-2003, 10:37 AM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	Where is sites.txt located? Line 347 in robot_functions.php is: PHP Code: `$path = $newurl["path"];` Could you change that to: PHP Code: `$path = $newurl["path"]; echo $path . " <--- Path<br><br>";` to see what $path is? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

10-06-2003, 03:51 PM	#6
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hmm... How about try the following: In the config file, set the following: PHP Code: `define('PHPDIG_SESSID_REMOVE',true); // remove SIDS from indexed URLS define('PHPDIG_SESSID_VAR','mscssid'); // name of the SID variable` Anyone out there ever have problems indexing *.asp files? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
spidering error = theURL,winName,features	ddowdall	Troubleshooting	0	03-19-2006 08:28 AM
error while spidering	hhsuser	Script Installation	1	11-16-2004 07:31 PM

10-02-2003, 09:27 AM	#3
jimigisme Green Mole Join Date: Sep 2003 Posts: 10	Example of sites.txt http://www.sitename.com http://www.sitename.com http://www.sitename.com http://www.sitename.com

10-06-2003, 09:03 AM	#5
jimigisme Green Mole Join Date: Sep 2003 Posts: 10	Thank you for your help, here is what I got: SITE : http://www.welcometoourworldofgifts.com/ Exclude paths : - @NONE@ /welcome/default.asp <--- Path<br><br>+1:http://www.welcom (time : 00:00:28) /welcome/default.asp <--- Path<br><br>/welcome/default.asp