robots.txt not fully honored

[Carl Mikkelsen]

I have been deploying phpdig as a test or our intranet. Aside from encountering a php segmentation violation when parsing a cookie, the largest problem I've had is with processing of robots.txt.

Our intranet is almost entirely dynamic content -- much of what I want to index is delivered by TWiki, a collaboration tool distributed from www.twiki.org. In robot_functions.php, a test is made for each URL encountered to determine if it should be indexed. The logic for this test causes an HTTP HEAD request to be issued, I think to determine the content type. This HEAD request is issued without regard to the robots.txt file.

If the content type is appropriate, the robots.txt defined exclusions are tested.

Unfortunately, the HEAD request causes the content for a page to be generated. In some cases, that generation can be VERY lengthly. To all appearances, the indexing is stopped. The HTTP server grinds to a halt computing page content that is never used.

The "fix" I'm testing is to move the check of robots.txt to the beginning of the function. Iff the file is not excluded, then the content type can be tested as before.

If this is causing trouble for anyone else, and if the developers concur, I could post a patch.

[Charter]

Hi. Sure, post away. Also, you might be interested in this change.

[bakker]

I'm having a similar problem and would be interested in your patch.

[Carl Mikkelsen]

Attached is a patch file which could be applied to the robot_functions.php file includes with the phpdig distribution this morning.

This includes also changes to handle MSPOWERPOINT (which should be matched with declarations in includes/config.php), as well as some fixes inherited from an alternate robot_functions.php I downloaded.

There is one change where the DOMAIN field in cookies was causing php to crash. I removed the DOMAIN processing (without understanding the intent) which could cause problems. You can back out that change.

The main change is to move the robots.txt processing earlier, so that the http HEAD request is not performed.

I also fixed what seemed to this php novice to be problems escaping some characters in the robots.txt parsing. With this change, phpdig accepts "*" as a meta-character in robots.txt, allowing entries such as:
Disallow: dynamic-content/view*parm=

As I am unfamiliar with php, I'm asking both for php-related feedback, and for comments related to the intent of the changes.

Thanks,

-- Carl