Quick continue after accident and regular spiderind time.

sf44 · 07-07-2004, 02:17 AM

Hi,
How to quick continue spidering/indexing after break?
My computer have crashed during spiderind/indexing proces. I am using Mozilla browser.
The site is locked - I m able to unlock it in admin, but what next ?
Update takes a lot of time, I just want to continue.
I am trying to spider more then 2000 pages ~ 30kB - I have download them to local computer.
But spidering sites takes too long - about 400 pages - 24 hours and it takes longer and longer.
Is it regular ?

Slavomir Slovakia.

sf44 · 07-07-2004, 09:03 AM

I have solved this problem. Is is nescessary to work with database.
After some testing I have found that at my computer script which run from console is about 10 times more quick then browser based.

Slavomir.

Charter · 07-07-2004, 03:59 PM

Hi. Here are some answers, although you may have already figured them out.

>> How to quick continue spidering/indexing after break?

If spidering ends prematurely, there may be index info remaining in the tempspider table. If so, just enter the URL again or choose the site and update.

>> The site is locked - I m able to unlock it in admin, but what next ?

See above.

>> But spidering sites takes too long - about 400 pages - 24 hours and it takes longer and longer.
Is it regular ?

Maybe update to version 1.8.1 or make sure that the tempspider table is empty between runs. To empty the tempspider table, just click the delete button without selecting a site.

>> Is is nescessary to work with database.

Some people using Win/IIS servers have had to fiddle with the database tables, but if you are using Apache, then it shouldn't be necessary to work directly with the tables.

>> After some testing I have found that at my computer script which run from console is about 10 times more quick then browser based.

Welcome to shell.

07-07-2004, 02:17 AM	#1
sf44 Green Mole Join Date: Jul 2004 Posts: 9	Quick continue after accident and regular spiderind time. Hi, How to quick continue spidering/indexing after break? My computer have crashed during spiderind/indexing proces. I am using Mozilla browser. The site is locked - I m able to unlock it in admin, but what next ? Update takes a lot of time, I just want to continue. I am trying to spider more then 2000 pages ~ 30kB - I have download them to local computer. But spidering sites takes too long - about 400 pages - 24 hours and it takes longer and longer. Is it regular ? Slavomir Slovakia. Last edited by sf44; 07-07-2004 at 02:20 AM.

07-07-2004, 03:59 PM	#3
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Here are some answers, although you may have already figured them out. >> How to quick continue spidering/indexing after break? If spidering ends prematurely, there may be index info remaining in the tempspider table. If so, just enter the URL again or choose the site and update. >> The site is locked - I m able to unlock it in admin, but what next ? See above. >> But spidering sites takes too long - about 400 pages - 24 hours and it takes longer and longer. Is it regular ? Maybe update to version 1.8.1 or make sure that the tempspider table is empty between runs. To empty the tempspider table, just click the delete button without selecting a site. >> Is is nescessary to work with database. Some people using Win/IIS servers have had to fiddle with the database tables, but if you are using Apache, then it shouldn't be necessary to work directly with the tables. >> After some testing I have found that at my computer script which run from console is about 10 times more quick then browser based. Welcome to shell. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
how to continue spidering when connection was been broken	zaartix	How-to Forum	7	12-09-2004 07:18 AM
do i need to use the radio buttons or just the regular expressions?	Fking	How-to Forum	1	10-04-2004 10:18 AM
quick search feature on index page	bigals	How-to Forum	3	04-02-2004 04:21 AM
Regular Expression Fix	Charter	Mod Submissions	0	12-02-2003 11:58 PM
Request Time	mongeon	Mod Requests	1	10-05-2003 10:15 AM

07-07-2004, 09:03 AM	#2
sf44 Green Mole Join Date: Jul 2004 Posts: 9	I have solved this problem. Is is nescessary to work with database. After some testing I have found that at my computer script which run from console is about 10 times more quick then browser based. Slavomir.