|
06-07-2004, 03:51 PM | #1 |
Green Mole
Join Date: Apr 2004
Location: Cali
Posts: 10
|
Rate of spidering: is it determined by the server?
I was wondering if anyone happned to know what would determine the speed of the spidering. Currently, I am spidering at an average rate of 375 URLs per hour. That seems rather slow. Would that have anything to do with the server's processor speeds? Or would it be a combination of a bunch of different factors such as:
1.) Server processor speed. 2.) Server OS. 3.) Internet bandwidth of server. 4.) My client script on my browser. I tend to think it's 1-3 and not #4. But if anyone else has some feedback about how fast they are able to spider I would appreciate it. Thanks. |
06-07-2004, 05:48 PM | #2 |
Purple Mole
Join Date: Jan 2004
Posts: 694
|
When I spidered my site recently around noon that day, it took over 2 hours to spider about 1,500 pages. I re-spidered the same site close to midnight that same day, and it took about 45 minutes. I tend to think that during lighter traffic times on your website, the spidering process would be faster. Just a guess though.
|
06-08-2004, 01:56 AM | #3 |
Green Mole
Join Date: Mar 2004
Posts: 22
|
1.) Server processor speed.
Of course this will change indexing speed 2.) Server OS. Linux should be faster (it is for MySQL) 3.) Internet bandwidth of server. If you index site that aren't on the server, of course it mather 4.) My client script on my browser. No, this should't change anything 5) PHP configuration If you have a low memory limit and so on it can slow indexing process 6)Load of your server If there's ressource intensive scripts on your server this can also be scripts of your neighbours (if you're on a shared server) this can slow down indexing. Try to know where your server is located (I mean lot of european server are located in USA) to choose the right hour to do the job. |
06-08-2004, 10:53 AM | #4 |
Green Mole
Join Date: Apr 2004
Location: Cali
Posts: 10
|
Hmmm. Thanks. It makes sense.
Thanks, guys. It's very much appreciated. I had a feeling that there would be a few contributing factors. I bet my server is pretty bogged down since the number of databases being used.
In the future I suppose I will have to consider renting my own server somewhere. If anyone knows of any great rates with PHP 4.3+ and MySQL I'd appreciate it. Otherwise I was thinking about a local server company, http://www.serverbeach.com which I believe has a good rate ($99/month) for Linux Redhat. But I'm still debating about this. Again, thanks. I really appreciate the info. I'll have to do some more brainstorming about what would be the best thing to do. |
06-08-2004, 05:44 PM | #5 |
Purple Mole
Join Date: Jan 2004
Posts: 694
|
You don't say how much disk space you require. That makes a big difference in what web host anyone could recommend. My web host is MindStormHosting. I've been with them for about six months and have been very happy with their service. There are several hosting packages to choose from. You might want to check them out to see if they'd have what you need.
|
06-08-2004, 06:20 PM | #6 | |
Green Mole
Join Date: Apr 2004
Location: Cali
Posts: 10
|
Hi Vinyl, thanks!
Quote:
I am not sure how much storage space I would need however, I am looking to grow a phpdig-based website in terms of collecting as many urls as possible but am currently on a limited budget, so I really do not know as of yet. However, more of anything in terms of hardware and software would always be better, me thinks. |
|
06-09-2004, 05:39 AM | #7 |
Green Mole
Join Date: Jun 2004
Posts: 3
|
Spidering own server: optimization?
I was wondering if there are any optimizations one can make when spidering a site hosted on the same server (same domain)? In particular, if I tell phpdig to spider www.mydomain.com, doesn't this involve the DNS server and a roundtrip to the internet? I tried localhost, but that didn't work (shared hosting). Any suggestions?
__________________
-Rob ------ visit me at www.robshouse.net |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Spider indexing/request speed/rate - How to change? | JAB Creations | How-to Forum | 1 | 09-07-2005 05:46 AM |
Moving to a new server | ezytrak | How-to Forum | 1 | 03-03-2005 08:11 AM |
Hello, I use a Windows Server | ClausBrell | The Mole Hole | 2 | 09-30-2004 04:35 AM |
Test Server | RaGe | Mod Requests | 0 | 05-10-2004 04:01 PM |
Spidering Problems on a Windows Server Website | vinyl-junkie | Troubleshooting | 23 | 02-20-2004 06:44 PM |