|
Download -
PHP Reference -
Links -
Demo 1.8.9 RC1 -
API
License - Credits - Copying - Changelog - Documentation |
|
|
03-05-2004, 08:23 AM
|
#1 |
|
Head Mole
Join Date: May 2003
Posts: 2,539
|
Using PhpDig on this site
Hi. Most 'big player' robots are allowed to crawl this site. However, please do not use PhpDig to try and crawl this site. Any income currently earned from this site does not cover the costs to operate this site. Please be courteous of this fact, when using a 'homemade' robot like PhpDig, before crawling this or other sites. Thanks.
xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:41 -0800] "HEAD /robots.txt HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)" xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:42 -0800] "GET /robots.txt HTTP/1.0" 200 1145 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)" xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:43 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)" xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:44 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
|
|
 |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| New phpDig indexes whole site now or? | JAB Creations | The Mole Hole | 0 | 11-07-2007 03:14 AM |
| Switch of site with PHPDig to new server | leslief | Script Installation | 0 | 02-13-2007 08:19 AM |
| successful indexing of every site but site where phpdig is served | phillystyle123 | Troubleshooting | 1 | 02-21-2005 09:06 PM |
| phpdig blocked when spidering any site | heli | Troubleshooting | 3 | 09-30-2004 10:42 AM |
| Can't get PHPDig to index an htaccess protected site | mlerch@mac.com | Troubleshooting | 28 | 02-25-2004 03:13 PM |
