Set time limit on spider.php or number of pages

paullind · 04-26-2004, 05:09 PM

Hi

I use a commercial server which has limits to the amount of resources I can use with spider.php. They've warned me before about the script taking up too many resources and are threatening to shut me down. I usually give spider.php a list of 30 sites (level0 spidering depth) to spider at a time as a cron job

Can I limit spider.php to run a maximim of 10 minutes even if it hasn't completed a cron job on a list of url's, to help prevent my site from being shut down by an angry administrator?

Thx

Paul L

Charter · 05-01-2004, 08:25 AM

Hi. If your host allows, set a cron job like so:

Code:

10 * * * * ps -ef | grep 'php -f spider.php' | awk '{print $2}' | xargs kill -9

10 * * * *               : complete this line to be 10 minutes after spider start
ps -ef                   : list out the processes
grep 'php -f spider.php' : find the spider process
awk '{print $2}'         : get PID from column two
xargs kill -9            : kill the spider process

04-26-2004, 05:09 PM	#1
paullind Orange Mole Join Date: Jan 2004 Posts: 30	Set time limit on spider.php or number of pages Hi I use a commercial server which has limits to the amount of resources I can use with spider.php. They've warned me before about the script taking up too many resources and are threatening to shut me down. I usually give spider.php a list of 30 sites (level0 spidering depth) to spider at a time as a cron job Can I limit spider.php to run a maximim of 10 minutes even if it hasn't completed a cron job on a list of url's, to help prevent my site from being shut down by an angry administrator? Thx Paul L

05-01-2004, 08:25 AM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. If your host allows, set a cron job like so: Code: 10 * * * * ps -ef \| grep 'php -f spider.php' \| awk '{print $2}' \| xargs kill -9 10 * * * * : complete this line to be 10 minutes after spider start ps -ef : list out the processes grep 'php -f spider.php' : find the spider process awk '{print $2}' : get PID from column two xargs kill -9 : kill the spider process __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
set time limit	Macfplus	Script Installation	0	11-17-2006 09:03 AM
Number of pages indexed	claudiomet	How-to Forum	0	08-30-2004 03:26 PM
I get a warning - Cannot set time limit!!	macin1818	Troubleshooting	1	04-20-2004 11:51 AM
Spider & time limit	onlytrue	How-to Forum	1	04-16-2004 06:03 AM
Limit number of spidered pages	Not Logged In	How-to Forum	5	12-16-2003 04:03 PM