robots.txt and URL

djavet · 01-10-2005, 11:53 PM

Hello,

I've a robot.txt file wich work perfectly with PhpDig.
Notice the forum folder into:

Code:

User-agent: *
Disallow: /cgi-bin/
Disallow: /flash/
Disallow: /forum/

Into my forum (PhpBB), i've a FAQ section wich I'd index with PhpDig.
It is possible to force a few URLs (in fact 19 entries) when the forum is exclude form indexing?

Thx for your help and time.
Regards; Dominique

Dave A · 01-11-2005, 12:27 AM

From what I can see your are asking how to make PHP spider a directory that has been excluded in the robots.txt file.
From my experience of the software it is built into PHPDIG to read the robots.txt file and obey it, ethically if the web master has excluded the directory from robots would it be right to try and index it?
No doubt people with more experience than me may well answer your question or know how it's done but I would ask the question "Wouldn't that make the robot spider little more than a hacking device?"
Hopefuly someone with more knowledge will help you.

djavet · 01-11-2005, 01:18 AM

No ambigous way in my question, but I understand what you mean and I've not think about until you talk about.
I've a forum (www.john-howe.com/forum) with a lot of section and one is about FAQ wich I wisch to include into indexing.
My question is: How can I do that?
I don't want to list into robots.txt my thousand treads

It is possible to specified at robot.txt wich URL index? I've found nothing about. Is "Allow:" supported in phpdig?

Regards, Dom

Charter · 01-11-2005, 03:05 AM

First, remove "Disallow: /forum/" from your robots.txt file.

Next, go to the PhpDig admin panel and copy down the "update sites" values for your site.

Then, enter each FAQ-type URL that you want to index, one per line, in the PhpDig textbox like so:

Code:

http://www.john-howe.com/forum/phpbb/viewtopic.php?t=XXXX
http://www.john-howe.com/forum/phpbb/viewtopic.php?t=YYYY
http://www.john-howe.com/forum/phpbb/viewtopic.php?t=ZZZZ

Now, set the radio button to no, search depth to zero, links per to zero, and click the dig button.

Once PhpDig is done, go to "update sites" and edit the values back to their original settings.

Remember to add "Disallow: /forum/" back to your robots.txt file.

PhpDig currently does not understand robots.txt "allow" lines.

Also, read this documentation for further information.

djavet · 01-11-2005, 04:19 AM

Super tricks. Thx a lots.
I will try it tonigh.

Regards, Dom

01-10-2005, 11:53 PM	#1
djavet Orange Mole Join Date: Jan 2005 Posts: 31	robots.txt and URL Hello, I've a robot.txt file wich work perfectly with PhpDig. Notice the forum folder into: Code: User-agent: * Disallow: /cgi-bin/ Disallow: /flash/ Disallow: /forum/ Into my forum (PhpBB), i've a FAQ section wich I'd index with PhpDig. It is possible to force a few URLs (in fact 19 entries) when the forum is exclude form indexing? Thx for your help and time. Regards; Dominique

01-11-2005, 01:18 AM	#3
djavet Orange Mole Join Date: Jan 2005 Posts: 31	No ambigous way in my question, but I understand what you mean and I've not think about until you talk about. I've a forum (www.john-howe.com/forum) with a lot of section and one is about FAQ wich I wisch to include into indexing. My question is: How can I do that? I don't want to list into robots.txt my thousand treads It is possible to specified at robot.txt wich URL index? I've found nothing about. Is "Allow:" supported in phpdig? Regards, Dom Last edited by djavet; 01-11-2005 at 01:37 AM.

01-11-2005, 03:05 AM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	First, remove "Disallow: /forum/" from your robots.txt file. Next, go to the PhpDig admin panel and copy down the "update sites" values for your site. Then, enter each FAQ-type URL that you want to index, one per line, in the PhpDig textbox like so: Code: http://www.john-howe.com/forum/phpbb/viewtopic.php?t=XXXX http://www.john-howe.com/forum/phpbb/viewtopic.php?t=YYYY http://www.john-howe.com/forum/phpbb/viewtopic.php?t=ZZZZ Now, set the radio button to no, search depth to zero, links per to zero, and click the dig button. Once PhpDig is done, go to "update sites" and edit the values back to their original settings. Remember to add "Disallow: /forum/" back to your robots.txt file. PhpDig currently does not understand robots.txt "allow" lines. Also, read this documentation for further information. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
robots.txt seems to be ignored :?	galacticvoyager	Bug Tracker	1	11-12-2005 01:52 PM
robots.txt comments	edkay	Mod Submissions	2	03-12-2004 01:41 PM
robots.txt versus robotsxx.txt	Charter	IPs, SEs, & UAs	0	03-11-2004 07:00 PM
robots.txt ignored	roy	Troubleshooting	3	02-20-2004 09:02 PM
robots.txt	renehaentjens	Troubleshooting	3	12-05-2003 03:40 PM

01-11-2005, 12:27 AM	#2
Dave A Purple Mole Join Date: Aug 2004 Location: North Island New Zealand Posts: 170	From what I can see your are asking how to make PHP spider a directory that has been excluded in the robots.txt file. From my experience of the software it is built into PHPDIG to read the robots.txt file and obey it, ethically if the web master has excluded the directory from robots would it be right to try and index it? No doubt people with more experience than me may well answer your question or know how it's done but I would ask the question "Wouldn't that make the robot spider little more than a hacking device?" Hopefuly someone with more knowledge will help you.

01-11-2005, 04:19 AM	#5
djavet Orange Mole Join Date: Jan 2005 Posts: 31	Super tricks. Thx a lots. I will try it tonigh. Regards, Dom