Multiple Spiders

jmitchell · 12-16-2004, 12:57 PM

Hi there,

I have just installed phpdig 2 days and I'm very excited about using it! I have been spidering for almost 2 days, and I would like to know, is there any way to run multiple spiders?

I have seen several threads asking the same question, and some of them had answers, but can I get some tested code that is being used currently for multiple spiders?

thanks

J Mitchell

Charter · 12-16-2004, 03:15 PM

Maybe you can get an idea from this thread or try using multiple browser windows.

jmitchell · 12-16-2004, 04:52 PM

so, I can have multiple browsers on multiple pcs running the spider at the same time and it will work?

Charter · 12-16-2004, 05:43 PM

If you are going to use multiple browsers, then you might want to comment out the following line in the spider.php file:

PHP Code:


			
array_unshift($list_sites,$site_datas);

If you are going to use shell, then you might want to send in only one site per process.

Either way, you should test it out on some of your demo sites and see if it works, whether you need to do other modifications to the code, etcetera.

12-16-2004, 12:57 PM	#1
jmitchell Orange Mole Join Date: Dec 2004 Location: Tennessee Posts: 60	Multiple Spiders Hi there, I have just installed phpdig 2 days and I'm very excited about using it! I have been spidering for almost 2 days, and I would like to know, is there any way to run multiple spiders? I have seen several threads asking the same question, and some of them had answers, but can I get some tested code that is being used currently for multiple spiders? thanks J Mitchell

12-16-2004, 03:15 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Maybe you can get an idea from this thread or try using multiple browser windows. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

12-16-2004, 05:43 PM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	If you are going to use multiple browsers, then you might want to comment out the following line in the spider.php file: PHP Code: `array_unshift($list_sites,$site_datas);` If you are going to use shell, then you might want to send in only one site per process. Either way, you should test it out on some of your demo sites and see if it works, whether you need to do other modifications to the code, etcetera. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
wrapper/multiple spiders...	CentaurAtlas	Mod Submissions	5	11-19-2006 01:07 PM
Locked one of my spiders?	megabaz	Troubleshooting	4	10-23-2004 05:37 PM
1.8.3 spiders slow, 1.6.3 spiders same site fast	Wayne McBryde	Troubleshooting	0	09-21-2004 08:10 PM
Multiple spiders	tryangle	How-to Forum	3	04-24-2004 03:43 AM
2 or more Spiders	firestarter	Mod Requests	2	03-29-2004 12:59 PM

12-16-2004, 04:52 PM	#3
jmitchell Orange Mole Join Date: Dec 2004 Location: Tennessee Posts: 60	so, I can have multiple browsers on multiple pcs running the spider at the same time and it will work?