PhpDig.net - View Single Post - Calling all persons spidering multiple domains

Slider · 11-02-2004, 09:12 AM

I would like to hear from different people using PhpDig that use it to crawl many different domains.

1. How many sites maximum have you spidered with PhpDig?
My curiosity is a result of wanting to have 10,000 sites listed at max and need to know if PhpDig can handle this or even what problems I might run into.

2. At your most sites crawled, how long does it take for crawling to finish?

3. Do you find that you run into any problems with non-relevant results and have to work to refine searches?

4. Any additonal information about problems that I may incur would be helpful as well.

I thank you in advance for your time and contributions to this thread.
David

11-02-2004, 09:12 AM	#1
Slider Orange Mole Join Date: Jan 2004 Posts: 30	Calling all persons spidering multiple domains I would like to hear from different people using PhpDig that use it to crawl many different domains. 1. How many sites maximum have you spidered with PhpDig? My curiosity is a result of wanting to have 10,000 sites listed at max and need to know if PhpDig can handle this or even what problems I might run into. 2. At your most sites crawled, how long does it take for crawling to finish? 3. Do you find that you run into any problems with non-relevant results and have to work to refine searches? 4. Any additonal information about problems that I may incur would be helpful as well. I thank you in advance for your time and contributions to this thread. David