Spider process does not do anything

acidelic · 04-27-2004, 10:58 AM

I'm encountering a problem getting the spider to work. Creating a robots.txt file did not seem to help.

Here is my robots.txt:

User-agent: *
Disallow: /include/

When I tell it to spider localhost or the hostname of the local server (depth=5) always results in this:

Spidering in progress...

--------------------------------------------------------------------------------
SITE : http://localhost/
Exclude paths :
- include/

There are no CPU cycles being used and the page is fully loaded. What else can I check? How will I know that the spider process is actually doing something?

Thanks

Charter · 05-01-2004, 08:32 AM

Hi. Try this thread.

acidelic · 05-10-2004, 10:26 AM

I took a look at the article and I don't think that is the problem. I have a single web server with one NIC. I'm trying to spider the local web server, but I never see any results. I don't see any progress on the page (which also says it is "Done") and it will sit there forever doing nothing. I tried spidering some other sites but nothing happens with those either.

What else is there that I can check?

Siava · 05-12-2004, 03:33 AM

acidelic
In config.php set up define('USE_IS_EXECUTABLE_COMMAND','0');

(It is the best search script!)

acidelic · 05-12-2004, 05:48 AM

That did the trick!!! The spider page didn't load, but if I refresh the admin page I can see the counts going up. And I can actual search for things on the site. Thanks for your help Siava!

04-27-2004, 10:58 AM	#1
acidelic Green Mole Join Date: Apr 2004 Posts: 4	Spider process does not do anything I'm encountering a problem getting the spider to work. Creating a robots.txt file did not seem to help. Here is my robots.txt: User-agent: * Disallow: /include/ When I tell it to spider localhost or the hostname of the local server (depth=5) always results in this: Spidering in progress... -------------------------------------------------------------------------------- SITE : http://localhost/ Exclude paths : - include/ There are no CPU cycles being used and the page is fully loaded. What else can I check? How will I know that the spider process is actually doing something? Thanks

05-01-2004, 08:32 AM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Try this thread. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Spidering process	hanhand	Troubleshooting	1	07-04-2004 05:03 PM

05-10-2004, 10:26 AM	#3
acidelic Green Mole Join Date: Apr 2004 Posts: 4	I took a look at the article and I don't think that is the problem. I have a single web server with one NIC. I'm trying to spider the local web server, but I never see any results. I don't see any progress on the page (which also says it is "Done") and it will sit there forever doing nothing. I tried spidering some other sites but nothing happens with those either. What else is there that I can check?

05-12-2004, 03:33 AM	#4
Siava Green Mole Join Date: May 2004 Location: Russia (Saint-Petersbrg) Posts: 16	acidelic In config.php set up define('USE_IS_EXECUTABLE_COMMAND','0'); (It is the best search script!)

05-12-2004, 05:48 AM	#5
acidelic Green Mole Join Date: Apr 2004 Posts: 4	That did the trick!!! The spider page didn't load, but if I refresh the admin page I can see the counts going up. And I can actual search for things on the site. Thanks for your help Siava!