PhpDig.net - View Single Post - Rate of spidering: is it determined by the server?

misterbearcom · 06-07-2004, 03:51 PM

I was wondering if anyone happned to know what would determine the speed of the spidering. Currently, I am spidering at an average rate of 375 URLs per hour. That seems rather slow. Would that have anything to do with the server's processor speeds? Or would it be a combination of a bunch of different factors such as:

1.) Server processor speed.
2.) Server OS.
3.) Internet bandwidth of server.
4.) My client script on my browser.

I tend to think it's 1-3 and not #4. But if anyone else has some feedback about how fast they are able to spider I would appreciate it. Thanks.

06-07-2004, 03:51 PM	#1
misterbearcom Green Mole Join Date: Apr 2004 Location: Cali Posts: 10	Rate of spidering: is it determined by the server? I was wondering if anyone happned to know what would determine the speed of the spidering. Currently, I am spidering at an average rate of 375 URLs per hour. That seems rather slow. Would that have anything to do with the server's processor speeds? Or would it be a combination of a bunch of different factors such as: 1.) Server processor speed. 2.) Server OS. 3.) Internet bandwidth of server. 4.) My client script on my browser. I tend to think it's 1-3 and not #4. But if anyone else has some feedback about how fast they are able to spider I would appreciate it. Thanks.