Spider stops before all pages are indexed

halide · 07-09-2005, 10:21 AM

Hi,

I am trying to spider 3 sections of my site via a sitemap which I have put together for the purpose. The problem is that sometimes the spider will index all the pages but often it will stop before the end. There is no error message or completion message, just nothing. If I go back to the admin page it says *Locked* - This behaviour appears to be completely r****m because each time it stops at a different point.

The site is http://www.lightingdirectory.com and the site map is http://www.lightingdirectory.com/site-map (This is the URL I'm using for indexing).

Observation - All of the pages appear in the tempspider table every time. So it appears that the pages are being indexed but not spidered.

Any help would be appreciated. Thanks.

llamastephen · 07-15-2005, 05:59 AM

Did you manage to solve this? I am having very similar problems.

Charter · 07-18-2005, 05:38 PM

Once spidering is complete, the tempspider table should be empty. Check whether a server timeout issue is happening or whether the host is killing the spidering process.

halide · 07-19-2005, 01:26 AM

I haven't solved it yet. But when I set up the next crawl I'll also increase the time out value of my apache server and see whether that helps.

07-09-2005, 10:21 AM	#1
halide Green Mole Join Date: Jun 2005 Posts: 2	Spider stops before all pages are indexed Hi, I am trying to spider 3 sections of my site via a sitemap which I have put together for the purpose. The problem is that sometimes the spider will index all the pages but often it will stop before the end. There is no error message or completion message, just nothing. If I go back to the admin page it says Locked - This behaviour appears to be completely r****m because each time it stops at a different point. The site is http://www.lightingdirectory.com and the site map is http://www.lightingdirectory.com/site-map (This is the URL I'm using for indexing). Observation - All of the pages appear in the tempspider table every time. So it appears that the pages are being indexed but not spidered. Any help would be appreciated. Thanks.

07-18-2005, 05:38 PM	#3
Charter Head Mole Join Date: May 2003 Posts: 2,539	Once spidering is complete, the tempspider table should be empty. Check whether a server timeout issue is happening or whether the host is killing the spidering process. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Indexing stops after a few pages	Sibona	Troubleshooting	1	05-03-2005 11:27 AM
Spidering Stops, Site is Locked after 7 Pages	nevsie	Troubleshooting	3	03-02-2005 04:52 PM
Pages not re-indexed	wx3	Troubleshooting	0	09-16-2004 06:53 PM
Spidering stops after 6 level 1 pages	Steve	Troubleshooting	3	05-20-2004 05:56 PM
Spider stops indexing (Browser: Done)	rainbow_six	Troubleshooting	2	04-24-2004 09:16 AM

07-15-2005, 05:59 AM	#2
llamastephen Green Mole Join Date: Jul 2005 Posts: 3	Did you manage to solve this? I am having very similar problems.

07-19-2005, 01:26 AM	#4
halide Green Mole Join Date: Jun 2005 Posts: 2	I haven't solved it yet. But when I set up the next crawl I'll also increase the time out value of my apache server and see whether that helps.