|
Download -
PHP Reference -
Links -
Demo 1.8.9 RC1 -
API
License - Credits - Copying - Changelog - Documentation |
|
|
03-11-2004, 07:50 AM
|
#1 |
|
Green Mole
Join Date: Mar 2004
Posts: 20
|
Can I spider .asp pages?
SUrely an easy answer to this... but I can't find it!!!
I'm trying to crawl a website built entirely in asp, but it wont spider past the fron page. Please help! Thanks Gary |
|
|
|
03-11-2004, 02:26 PM
|
#2 |
|
Green Mole
Join Date: Mar 2004
Posts: 20
|
Why Can't I dig this site????
http://www.architecturecentre.net
digging results in: SITE : http://www.architecturecentre.net/ Exclude paths : - @NONE@ 1:http://www.architecturecentre.net/ (time : 00:00:06) No link in temporary table -------------------------------------------------------------------------------- links found : 1 http://www.architecturecentre.net/ Optimizing tables... Indexing complete ! There's loads of links on there!!!! ANy ideas? |
|
|
|
|
03-11-2004, 02:50 PM
|
#3 |
|
Head Mole
Join Date: May 2003
Posts: 2,539
|
Hi lighthouse, and welcome to PhpDig.net!
If you uncomment the following line in the robot_functions.php file, you'll see that the site/server sends back a "HTTP/1.1 403 Forbidden" error. PHP Code:
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
|
|
|
|
03-12-2004, 01:05 AM
|
#4 |
|
Green Mole
Join Date: Mar 2004
Posts: 20
|
Hi Charter!!!
Thanks for replying.... So basically this site itself won't let me spider it?... What would the site admin of this site have to do to allow me access? Thanks |
|
|
|
|
03-12-2004, 01:09 AM
|
#5 |
|
Head Mole
Join Date: May 2003
Posts: 2,539
|
>> So basically this site itself won't let me spider it?
Hi, yep that's it. The 403 means forbidden, not allowed, go away. >> What would the site admin of this site have to do to allow me access? Really, I don't have much of an idea as it's not my server, but my guess is to allow HEAD requests.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
|
|
|
 |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| dig for certain words | nmott | How-to Forum | 1 | 02-26-2005 09:02 PM |
| Search Engines and Dig | new2dev | How-to Forum | 1 | 02-21-2005 02:19 PM |
| is 'dig 1.8.3 slower than 'dig 1.8.0 | JÿGius³ | Troubleshooting | 1 | 08-02-2004 08:32 PM |
| PHP dig not indexing site on clean install... | mixonic | Troubleshooting | 1 | 06-28-2004 09:15 AM |
| how to dig only 1 page | zaartix | Troubleshooting | 8 | 05-11-2004 02:23 AM |
