cannot spider own site - others yes

web newsroom · 05-23-2004, 04:28 PM

Im sure that its something quite simple.

I have been able to successfully spider certain sites and currently show the following stats.

DataBase status
Hosts 140
Pages 7622
Index 2007006
Keywords 170156
Excluded paths 101
Temporary table 143915
Log entries 643

However, I still cannot seem to spider our own root site.
This seems top have been a problem for others posting here as well. Is it a domain name problem? Im not sure. I have complete control over my dns and it work correctly but still cannot spider our root site

A qualified subdomain.mydamain.com will work? I have changed the robots.txt and the .htaccess file and still stumped.

Any help would be appreciated

__________________
http://www.thewebnewsroom.com

web newsroom · 05-23-2004, 04:32 PM

actually just figured it out. pretty simple too.

Seems that if I delete the robots.txt it will allow me to spider.

So ill go back and delete the directories I dont want indexed and then replace the tobots.txt

thanks

05-23-2004, 04:28 PM	#1
web newsroom Green Mole Join Date: May 2004 Posts: 4	cannot spider own site - others yes Im sure that its something quite simple. I have been able to successfully spider certain sites and currently show the following stats. DataBase status Hosts 140 Pages 7622 Index 2007006 Keywords 170156 Excluded paths 101 Temporary table 143915 Log entries 643 However, I still cannot seem to spider our own root site. This seems top have been a problem for others posting here as well. Is it a domain name problem? Im not sure. I have complete control over my dns and it work correctly but still cannot spider our root site A qualified subdomain.mydamain.com will work? I have changed the robots.txt and the .htaccess file and still stumped. Any help would be appreciated __________________ http://www.thewebnewsroom.com __________________ http://www.thewebnewsroom.com

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Imcomplete spider : locked site but nothing append	beausej	Troubleshooting	1	07-18-2005 04:48 PM
Spider site with links	dell_10	External Binaries	10	09-20-2004 07:42 AM
spider only one site/file	jdc32	Troubleshooting	2	07-02-2004 06:49 AM
Help Please: Problems to spider... just one site	1001studio	Troubleshooting	1	06-28-2004 05:41 PM
spider entire site	summitmn	Troubleshooting	1	05-03-2004 05:30 PM

05-23-2004, 04:32 PM	#2
web newsroom Green Mole Join Date: May 2004 Posts: 4	actually just figured it out. pretty simple too. Seems that if I delete the robots.txt it will allow me to spider. So ill go back and delete the directories I dont want indexed and then replace the tobots.txt thanks