|
03-31-2004, 12:18 PM | #1 |
Green Mole
Join Date: Mar 2004
Posts: 6
|
How to change?
I have a large site about 800 pages but everytime i index it stops about 70 pages in. I noticed while it was doing the index that it is using
admin/spider.php?site_id=2&mode=small Is there a way to change the mode from small to something else so that i can index the whole site? |
04-01-2004, 12:03 AM | #2 |
Head Mole
Join Date: May 2003
Posts: 2,539
|
Hi. Do you mean that when it stops it isn't completely done with the index, or do you mean that it only indexes 70 of the 800 pages?
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
04-01-2004, 04:04 AM | #3 |
Green Mole
Join Date: Mar 2004
Posts: 6
|
yea when it stops it not done with the index i have to go back in a restart it and tell it to index again and it will get about 40 more pages then stop again and i have to keep redoing this
|
04-01-2004, 04:19 AM | #4 |
Head Mole
Join Date: May 2003
Posts: 2,539
|
Hi. This sounds like an issue possibly related to timeout or memory. What do you get in your error logs?
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
04-01-2004, 04:31 AM | #5 |
Green Mole
Join Date: Mar 2004
Posts: 6
|
the error logs that i have access to do not show timeouts or memory issues they are very basic logs and i can not access the other logs cause my host does not have them setup
|
04-01-2004, 04:34 AM | #6 |
Green Mole
Join Date: Mar 2004
Posts: 6
|
when i go back and tell it to respider the root it will go through all the previous pages saying date unchanged which is fine and then it will start to add about 40 or so more pages after it checks all the ones it did before. I got it to about 400 pages so far and it runs for about 40 minutes so i dont think that its a timeout error.
|
04-01-2004, 05:06 AM | #7 |
Head Mole
Join Date: May 2003
Posts: 2,539
|
Hi. Do you notice whether or not it only stops when it encounters a large file? If so, there is some code in this thread that might help.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
04-01-2004, 05:08 AM | #8 |
Green Mole
Join Date: Mar 2004
Posts: 6
|
all of my files are small they are basic web pages with text most are about 300k or less. i will try that code tho just to make sure.
|
04-01-2004, 05:19 AM | #9 |
Head Mole
Join Date: May 2003
Posts: 2,539
|
Hi. Could it be that your host has some sort of auto shutdown for runaway or resource intensive scripts?
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
04-01-2004, 05:26 AM | #10 |
Green Mole
Join Date: Mar 2004
Posts: 6
|
it may be that cause thats what i am starting to think. so i guess i will just index the pages about 50 at a time. Once i get all the pages indexed it reindexes them all fine. Like right now i have gone though and reindexed the root enough times to get about 500 pages now and it goes though all the previuos pages without a problem so i guess once i get all the pages indexed everything should be ok when it does the automatic reindex every week.
|
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Change POST to GET | esconsult1 | Mod Requests | 0 | 11-09-2005 03:56 AM |
Change sleep | noel | How-to Forum | 1 | 10-27-2005 09:58 AM |
Change Identity of Spider | Kishin | How-to Forum | 1 | 07-18-2005 04:03 PM |
how to change user-agent ? | 123av | Mod Requests | 11 | 04-21-2004 04:29 AM |