PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 11-02-2004, 09:12 AM   #1
Slider
Orange Mole
 
Join Date: Jan 2004
Posts: 30
Post Calling all persons spidering multiple domains

I would like to hear from different people using PhpDig that use it to crawl many different domains.

1. How many sites maximum have you spidered with PhpDig?
My curiosity is a result of wanting to have 10,000 sites listed at max and need to know if PhpDig can handle this or even what problems I might run into.

2. At your most sites crawled, how long does it take for crawling to finish?

3. Do you find that you run into any problems with non-relevant results and have to work to refine searches?

4. Any additonal information about problems that I may incur would be helpful as well.

I thank you in advance for your time and contributions to this thread.
David
Slider is offline   Reply With Quote
Old 11-05-2004, 06:20 PM   #2
AllKnightAccess
Green Mole
 
Join Date: Sep 2004
Posts: 25
I am interested in this information too.
AllKnightAccess is offline   Reply With Quote
Old 11-06-2004, 01:12 AM   #3
Dave A
Purple Mole
 
Dave A's Avatar
 
Join Date: Aug 2004
Location: North Island New Zealand
Posts: 170
Judging by my results a lot would depend on how deep you spider the other domains because the MSQL file on your host would grow quite large and as an estimate, I would imagine that around 500 domains would need around 40mb of MYSQL space to store the data, plus a few megabytes of space for the files and that is based on spidering to a depth of around three, picking up say ten linked files per domain.
Spidering speed varies a lot depending on the quality of the host that your spidering and it can change because of the amount of data flying around the web.
In one of the other forums a guy is offering a prebuilt database so you could contact him, to find out the results he has got from using PHPDIG to spider multiple domains he may well have a good idea what kind of sizes and storage space you may need and PHPDIG's speed over heaps of domains.
PHPDIG as software really is brilliant and does what it is supposed to do and the help and support offered in these forums is brilliant.
I hope that this helps you..
Many regards
Dave A
Dave A is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Limit of spidering external domains Vadim How-to Forum 0 11-17-2006 09:53 AM
Banned Domains JLutterklas How-to Forum 0 09-05-2006 10:38 AM
Blocking domains richwilson How-to Forum 0 03-29-2006 06:02 AM
Spidering multiple URL's 2wheelin Mod Requests 0 05-22-2004 05:51 PM
Working with Domains bazarin How-to Forum 1 02-28-2004 03:28 PM


All times are GMT -8. The time now is 02:57 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.