PhpDig.net - Set time limit on spider.php or number of pages

PhpDig.net (http://www.phpdig.net/forum/index.php)

- Troubleshooting (http://www.phpdig.net/forum/forumdisplay.php?f=22)

- - Set time limit on spider.php or number of pages (http://www.phpdig.net/forum/showthread.php?t=872)

Set time limit on spider.php or number of pages

Hi

I use a commercial server which has limits to the amount of resources I can use with spider.php. They've warned me before about the script taking up too many resources and are threatening to shut me down. I usually give spider.php a list of 30 sites (level0 spidering depth) to spider at a time as a cron job

Can I limit spider.php to run a maximim of 10 minutes even if it hasn't completed a cron job on a list of url's, to help prevent my site from being shut down by an angry administrator?

Thx

Paul L

Hi. If your host allows, set a cron job like so:

Code:

10 * * * * ps -ef | grep 'php -f spider.php' | awk '{print $2}' | xargs kill -9



10 * * * *               : complete this line to be 10 minutes after spider start

ps -ef                   : list out the processes

grep 'php -f spider.php' : find the spider process

awk '{print $2}'         : get PID from column two

xargs kill -9            : kill the spider process