Can I make the spider stop and start on a dime?

dewed · 11-25-2005, 12:54 PM

I have a specific 36 hour window every week I'm allowed to spider a remote 300k+ page catalog site. I had been using wget in recursive mode, but I have no good way to stop it and restart where it left off the next week. I also have my own format I'd like the data stored in the mysql table.

Can phpdig be bent to meet my needs? or am I better off writing my own using curl and a very large database of urls to crawl, driven by a bash/perl/php script frontend?

11-25-2005, 12:54 PM	#1
dewed Green Mole Join Date: Nov 2005 Posts: 1	Can I make the spider stop and start on a dime? I have a specific 36 hour window every week I'm allowed to spider a remote 300k+ page catalog site. I had been using wget in recursive mode, but I have no good way to stop it and restart where it left off the next week. I also have my own format I'd like the data stored in the mysql table. Can phpdig be bent to meet my needs? or am I better off writing my own using curl and a very large database of urls to crawl, driven by a bash/perl/php script frontend?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How To stop spider by shell command ?	noel	How-to Forum	4	11-03-2005 02:06 PM
Only searching from start of word	benklocek	Troubleshooting	1	03-18-2005 02:14 PM
How can I make phpdig spider faster	jakeres	How-to Forum	1	11-29-2004 12:05 PM
Fixing spider.php, protecting from locking site after timeout or users stop	Konstantine	Mod Submissions	3	04-09-2004 01:37 PM
where do i start for installing this script	ekimbo	Script Installation	1	03-24-2004 11:41 PM