robots.txt comments

edkay · 03-02-2004, 05:54 AM

At present (1.8.0), the phpdigReadRobotsTxt function doesn't understand comments.

Robots.txt comments begin with a #. At present, these are ignored and so commented out lines are still interpretted (not good!!)

As a simple fix, I added the following additional check:

PHP Code:


			
while (list($id,$line) = each($robots))

            {

            // Ignore comments

            if (strpos(trim($line),"#") === 0)

                continue;

            if (ereg('^user-agent....

I would recommend that this (or similar) is included in future releases.

marb · 03-11-2004, 10:14 AM

In which file and where can I place this code?
I think it is robot_functions.php but I'm not sure.

Marten

Konstantine · 03-12-2004, 01:41 PM

Yes, it's robot_functions.php. Hope, it will be in new version of phpdig

03-02-2004, 05:54 AM	#1
edkay Green Mole Join Date: Feb 2004 Posts: 1	robots.txt comments At present (1.8.0), the phpdigReadRobotsTxt function doesn't understand comments. Robots.txt comments begin with a #. At present, these are ignored and so commented out lines are still interpretted (not good!!) As a simple fix, I added the following additional check: PHP Code: `while (list($id,$line) = each($robots)) { // Ignore comments if (strpos(trim($line),"#") === 0) continue; if (ereg('^user-agent....` I would recommend that this (or similar) is included in future releases.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
robots.txt seems to be ignored :?	galacticvoyager	Bug Tracker	1	11-12-2005 01:52 PM
robots.txt and URL	djavet	How-to Forum	4	01-11-2005 04:19 AM
robots.txt versus robotsxx.txt	Charter	IPs, SEs, & UAs	0	03-11-2004 07:00 PM
robots.txt ignored	roy	Troubleshooting	3	02-20-2004 09:02 PM
robots.txt	renehaentjens	Troubleshooting	3	12-05-2003 03:40 PM

03-11-2004, 10:14 AM	#2
marb Green Mole Join Date: Mar 2004 Posts: 19	In which file and where can I place this code? I think it is robot_functions.php but I'm not sure. Marten

03-12-2004, 01:41 PM	#3
Konstantine Green Mole Join Date: Mar 2004 Location: Russia Posts: 21	Yes, it's robot_functions.php. Hope, it will be in new version of phpdig