Can I spider .asp pages?
 

SUrely an easy answer to this... but I can't find it!!!

I'm trying to crawl a website built entirely in asp, but it wont spider past the fron page.

Please help!

Thanks
Gary

Why Can't I dig this site????
 

http://www.architecturecentre.net


digging results in:

SITE : http://www.architecturecentre.net/
Exclude paths :
- @NONE@
1:http://www.architecturecentre.net/
(time : 00:00:06)
No link in temporary table

--------------------------------------------------------------------------------

links found : 1
http://www.architecturecentre.net/
Optimizing tables...
Indexing complete !



There's loads of links on there!!!!

ANy ideas?

Hi lighthouse, and welcome to PhpDig.net!

If you uncomment the following line in the robot_functions.php file, you'll see that the site/server sends back a "HTTP/1.1 403 Forbidden" error.

Hi Charter!!!

Thanks for replying....
So basically this site itself won't let me spider it?...

What would the site admin of this site have to do to allow me access?

Thanks

>> So basically this site itself won't let me spider it?

Hi, yep that's it. The 403 means forbidden, not allowed, go away.

>> What would the site admin of this site have to do to allow me access?

Really, I don't have much of an idea as it's not my server, but my guess is to allow HEAD requests.