Thread: search box?
View Single Post
Old 08-18-2004, 10:55 PM   #3
WebDiva 2.0
Former Member
 
Join Date: Aug 2004
Posts: 18
Hi Yes, this did do the trick. I'm sorry, I'd just missed that in the config.php file as all the text was so "compact" together.

Put a few spaces in between lines of code, and ah, there it was staring me right in the eyes this whole time.

I have one other question, but didn't want to start another new thread just for that.

In the robot files, where the user-agent identifies itself, would this be appropriate (and proper protocol) to change the web URL identity to that of the domain actually running the spider query and also to change the name of the robot to something appropriate for the site running it?

Example: Site owner of some-domain.com decides he wants to run the phpDig script. The bot says to the site being queried, I am coming from phpdig.net my name is phpdig 1.8.3

When the truth is the bot is coming from some-domain.com
If the site owner wants to block his site from query and sets robots.txt Disallow: phpDig/ 1.8.3

these actions would cause all user of phpDig not to index the site whereas maybe just only one of the phpDig user sites are over indexing when others are not.

Better responsibility to robots.txt protocol suggests that the bots be easily identified by the source domain.

Therefore, some-domain.com would have bot called some-domain 1.8.3 with URI being some-domain.com

If wishing to be complaint with protocol, this needs be changed. Is doing this in some violation of phpDig user license?

If I am allowed to change these identifiers to be compliant with robots.txt protocol as outlined by the WC3 standards, where in the script would the changes need to be applied?

Would changing these affect any of the variables needed for running the script through cron or shell command via SSH? If so, where would changes also need to be made to accommodate these areas?
WebDiva 2.0 is offline   Reply With Quote