PhpDig.net - Spidering through a script

PhpDig.net (http://www.phpdig.net/forum/index.php)

- How-to Forum (http://www.phpdig.net/forum/forumdisplay.php?f=33)

- - Spidering through a script (http://www.phpdig.net/forum/showthread.php?t=312)

bloodjelly

12-19-2003 10:45 AM

Spidering through a script

Hi -

I'm trying to run the spider in the "background" through a php script, and I have this so far:

PHP Code:


		
			
$GLOBALS['limit'] = 2;

$GLOBALS['url'] = "http://www.website.com/";

include '../search/admin/spider.php';

This seems to run the spider fine for the appropriate website, and the website is entered into the MySQL database, but searching doesn't work on it until I run an update manually.

Is there a better way to run the spider without having to directly enter in the site to be spidered? Thanks!

Charter

12-19-2003 11:58 AM

Hi. You can run PhpDig from shell. Just set the following to the wanted search depth in the config file, make a text file with the full URLs, one per line, and use the below command.

PHP Code:


		
			
define('SPIDER_MAX_LIMIT',2);           //max recurse levels in sipder

define('SPIDER_DEFAULT_LIMIT',2);       //default value

define('RESPIDER_LIMIT',2);             //recurse limit for update

Code:

php -f [PHPDIG_DIR]/admin/spider.php [File containing an urls list]

More shell indexing options can be found here.

bloodjelly

12-19-2003 05:37 PM

This will work well, but is there a way to do it through a PHP script? I basically want "Install.php" to do x, y and z, plus get phpDig to spider a site, all in one execution without any user input. Can't I just feed spider.php the information it needs to spider the site? Thanks for the quick reply.

durr...just remembered I can do it with exec (/usr/bin/php -f spider.php); works well!

bloodjelly

03-15-2004 11:56 AM

OK - I used this command to spider:

PHP Code:


		
			
exec("/usr/bin/php -f /path/to/spider.php $site >> /dev/null &");

where $site = http://www.mysite.com/

This worked great until I recently upgraded to a newer version of PHP, and now the command doesn't produce any results. I know there's a way to turn on/off PHP as an executable, but I can't find it in PHP.INI and I'm not sure where to look. Thanks for helping out.

All times are GMT -8. The time now is 01:42 AM.