Calling all persons spidering multiple domains

Slider · 11-02-2004, 10:12 AM

I would like to hear from different people using PhpDig that use it to crawl many different domains.

1. How many sites maximum have you spidered with PhpDig?
My curiosity is a result of wanting to have 10,000 sites listed at max and need to know if PhpDig can handle this or even what problems I might run into.

2. At your most sites crawled, how long does it take for crawling to finish?

3. Do you find that you run into any problems with non-relevant results and have to work to refine searches?

4. Any additonal information about problems that I may incur would be helpful as well.

I thank you in advance for your time and contributions to this thread.
David

AllKnightAccess · 11-05-2004, 07:20 PM

I am interested in this information too.

Dave A · 11-06-2004, 02:12 AM

Judging by my results a lot would depend on how deep you spider the other domains because the MSQL file on your host would grow quite large and as an estimate, I would imagine that around 500 domains would need around 40mb of MYSQL space to store the data, plus a few megabytes of space for the files and that is based on spidering to a depth of around three, picking up say ten linked files per domain.
Spidering speed varies a lot depending on the quality of the host that your spidering and it can change because of the amount of data flying around the web.
In one of the other forums a guy is offering a prebuilt database so you could contact him, to find out the results he has got from using PHPDIG to spider multiple domains he may well have a good idea what kind of sizes and storage space you may need and PHPDIG's speed over heaps of domains.
PHPDIG as software really is brilliant and does what it is supposed to do and the help and support offered in these forums is brilliant.
I hope that this helps you..
Many regards
Dave A

11-02-2004, 10:12 AM	#1
Slider Orange Mole Join Date: Jan 2004 Posts: 30	Calling all persons spidering multiple domains I would like to hear from different people using PhpDig that use it to crawl many different domains. 1. How many sites maximum have you spidered with PhpDig? My curiosity is a result of wanting to have 10,000 sites listed at max and need to know if PhpDig can handle this or even what problems I might run into. 2. At your most sites crawled, how long does it take for crawling to finish? 3. Do you find that you run into any problems with non-relevant results and have to work to refine searches? 4. Any additonal information about problems that I may incur would be helpful as well. I thank you in advance for your time and contributions to this thread. David

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Limit of spidering external domains	Vadim	How-to Forum	0	11-17-2006 10:53 AM
Banned Domains	JLutterklas	How-to Forum	0	09-05-2006 11:38 AM
Blocking domains	richwilson	How-to Forum	0	03-29-2006 07:02 AM
Spidering multiple URL's	2wheelin	Mod Requests	0	05-22-2004 06:51 PM
Working with Domains	bazarin	How-to Forum	1	02-28-2004 04:28 PM

11-05-2004, 07:20 PM	#2
AllKnightAccess Green Mole Join Date: Sep 2004 Posts: 25	I am interested in this information too.

11-06-2004, 02:12 AM	#3
Dave A Purple Mole Join Date: Aug 2004 Location: North Island New Zealand Posts: 170	Judging by my results a lot would depend on how deep you spider the other domains because the MSQL file on your host would grow quite large and as an estimate, I would imagine that around 500 domains would need around 40mb of MYSQL space to store the data, plus a few megabytes of space for the files and that is based on spidering to a depth of around three, picking up say ten linked files per domain. Spidering speed varies a lot depending on the quality of the host that your spidering and it can change because of the amount of data flying around the web. In one of the other forums a guy is offering a prebuilt database so you could contact him, to find out the results he has got from using PHPDIG to spider multiple domains he may well have a good idea what kind of sizes and storage space you may need and PHPDIG's speed over heaps of domains. PHPDIG as software really is brilliant and does what it is supposed to do and the help and support offered in these forums is brilliant. I hope that this helps you.. Many regards Dave A