why is a temptable so big? (phpdigtempspider)

tapete · 12-18-2003, 04:12 AM

Hiho.

my phpdigtempspider table takes 4.6MB. Is that ok ore did something go wrong with the spider?

And my phpdigengine table takes 8.1MB and my phpdigkeywords only 2.8.

My computer crashed some times when indexing (hardware) so I'm not shure if something gone wrong.

Charter · 12-20-2003, 07:57 PM

Hi. Each keyword is associated with an id in the keywords table. These ids are associated with the text files in the engine table. With a large engine table, it seems like there are a lot of the same keywords showing up in different pages. The number of rows in the sites table is how many sites were crawled. The number of rows in the spider table is how many pages were crawled.

The tempspider may contain rows when the spidering process is stopped prematurely. These rows may be removed when the associated site is reindexed. The tempspider table may also contain rows even after a site has been indexed. The latter is something that needs further investigation. In the meantime, just make a backup of your database, and then afterwards empty the tempspider table if you want.

12-18-2003, 04:12 AM	#1
tapete Green Mole Join Date: Dec 2003 Posts: 1	why is a temptable so big? (phpdigtempspider) Hiho. my phpdigtempspider table takes 4.6MB. Is that ok ore did something go wrong with the spider? And my phpdigengine table takes 8.1MB and my phpdigkeywords only 2.8. My computer crashed some times when indexing (hardware) so I'm not shure if something gone wrong.

12-20-2003, 07:57 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Each keyword is associated with an id in the keywords table. These ids are associated with the text files in the engine table. With a large engine table, it seems like there are a lot of the same keywords showing up in different pages. The number of rows in the sites table is how many sites were crawled. The number of rows in the spider table is how many pages were crawled. The tempspider may contain rows when the spidering process is stopped prematurely. These rows may be removed when the associated site is reindexed. The tempspider table may also contain rows even after a site has been indexed. The latter is something that needs further investigation. In the meantime, just make a backup of your database, and then afterwards empty the tempspider table if you want. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Restart spider and index urls in temptable	jerrywin5	Troubleshooting	1	04-06-2005 02:18 PM
Mid Google search or big brother?	Dave A	How-to Forum	3	08-16-2004 01:20 AM
Big dig database - spidering question	JWSmythe	How-to Forum	3	05-14-2004 11:52 PM