Indexing slow.... no, _really_ slow

[bluntman]

Hi all,

I'm spidering a Windows 2000 server based site with several thousand pages on it. The indexing seems to be going really slowly. I don't mean 3-4 seconds, as most everyone else seems to be calling slow. It is taking over 1 minute for each page to index.

The results are the same whether I run the spider from the Windows 2000 host or an external Linux machine. The Linux machine does seem to run better in other respects (spidering doesn't stop from the web interface, etc), but it still takes the same amount of time.

Does anyone have any idea how to troubleshoot this?

Thanks

[Wayne McBryde]

If you post the URL of the site, I will try spidering it with PHPDig versions 1.6.3 & 1.8.3 and post back the time per page to spider. Of course I don't plan to spider the several thousand pages you got but I will spider 10 to 20 pages to get an average.

I currently spider over 1000 sites for my database and some sites do spider slow like the page per minute you talked about. Others will spider faster in the range of 3 to 4 seconds per page. I always guessed the slow sites were slow because of the server they were running on.