Again, indexing speed

Richie · 01-18-2004, 01:31 PM

Hi,

i have an archive containing about 90.000 *.htm - files in a directory and about 800 index pages linking them. a quick calculation on the actual indexing speed and the number of files leads me to about 40 days indexing those files.

is there an opportunity to index the site faster? the server load is extraordinary low, so there seems to bee enough overhead, but i did not find any opportunity to raise spidering speed or to spider with more than one instance.

As there are a)index files, b)thread indexes and c)messages, i have to put depth to 2.

Any Ideas? is phpdig still useful on a site of this size or should i better search for an other search engine?

thx in advance,

Richie

Richie · 01-19-2004, 12:21 AM

- sorry -
ignore. It was the problem of the index pahes with a lot of links. Actually, phpdig spiders about a message 1-2 seconds, and i guess it could be finisted tomorrow morning.

01-18-2004, 01:31 PM	#1
Richie Green Mole Join Date: Jan 2004 Posts: 2	Again, indexing speed Hi, i have an archive containing about 90.000 *.htm - files in a directory and about 800 index pages linking them. a quick calculation on the actual indexing speed and the number of files leads me to about 40 days indexing those files. is there an opportunity to index the site faster? the server load is extraordinary low, so there seems to bee enough overhead, but i did not find any opportunity to raise spidering speed or to spider with more than one instance. As there are a)index files, b)thread indexes and c)messages, i have to put depth to 2. Any Ideas? is phpdig still useful on a site of this size or should i better search for an other search engine? thx in advance, Richie

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Spider indexing/request speed/rate - How to change?	JAB Creations	How-to Forum	1	09-07-2005 06:46 AM
Indexing are very slow, how can i speed up?	ossi01	Troubleshooting	2	05-06-2005 04:26 AM
How to increase indexing speed	sf44	How-to Forum	0	01-28-2005 04:57 AM
Anything to speed up spidering	jinkas	Mod Requests	0	08-25-2004 03:07 PM
Speed up phpdig?	Konstantine	Mod Requests	0	03-13-2004 11:31 PM

01-19-2004, 12:21 AM	#2
Richie Green Mole Join Date: Jan 2004 Posts: 2	- sorry - ignore. It was the problem of the index pahes with a lot of links. Actually, phpdig spiders about a message 1-2 seconds, and i guess it could be finisted tomorrow morning.