Long Time Spidering Crashes

mikiwiki · 10-24-2006, 12:29 AM

I am spidering a huge web with a big amount of data. Very often, the web spidering (launched from admin) ends unexpectedly ("Page not found") 1, 2 or 3 hours later... Is it normal? Can a phpdig process, for example, work 24h continously?

Thanks

Charter · 10-26-2006, 10:25 AM

If you are using the admin panel, and spidering via your browser, you might consider using shell instead, especially if you are spidering for hours. Not sure but it might be the case that the browser cannot handle that much text. Plus, spidering from shell should be faster.

10-24-2006, 12:29 AM	#1
mikiwiki Green Mole Join Date: Apr 2006 Posts: 4	Long Time Spidering Crashes I am spidering a huge web with a big amount of data. Very often, the web spidering (launched from admin) ends unexpectedly ("Page not found") 1, 2 or 3 hours later... Is it normal? Can a phpdig process, for example, work 24h continously? Thanks

10-26-2006, 10:25 AM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	If you are using the admin panel, and spidering via your browser, you might consider using shell instead, especially if you are spidering for hours. Not sure but it might be the case that the browser cannot handle that much text. Plus, spidering from shell should be faster. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Long Call help.	GunMuse	How-to Forum	0	07-26-2006 11:11 AM
Long post, possible issues, and maybe the data to solve some?	Paul D. Buck	Troubleshooting	4	02-03-2005 12:59 PM
How long?	sjSJ	How-to Forum	1	05-11-2004 07:09 PM
PDFs in directory listed as one long entry	motopsycho	External Binaries	4	03-09-2004 06:07 PM
Request Time	mongeon	Mod Requests	1	10-05-2003 11:15 AM