PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Troubleshooting

Reply
 
Thread Tools
Old 05-14-2004, 12:38 AM   #1
Siava
Green Mole
 
Join Date: May 2004
Location: Russia (Saint-Petersbrg)
Posts: 16
Indexing local phpbb forum

Hello to all!

I have any problems with indexing my local phpbb forum.
I create robots.txt, include into this file
Disallow: /forum/posting.php
and other php files, BUT during the spidering I see that this files successfully indexing and generate the pages with this files!
robots.txt use a Mac encoding, file robot_function is updated, robots.txt is locate in root directory of the site.

(sorry for my not good english)
Siava is offline   Reply With Quote
Old 05-14-2004, 03:23 AM   #2
Pulsar-san
Green Mole
 
Join Date: May 2004
Location: France
Posts: 8
Your robots.txt is not in the root of your website, when typing
hrrp://siava.spb.ru/robots.txt I get a 404 page instead of getting the listing of the robots.txt file.
Pulsar-san is offline   Reply With Quote
Old 05-14-2004, 03:24 AM   #3
vinyl-junkie
Purple Mole
 
Join Date: Jan 2004
Posts: 694
Your robots.txt needs to look like this:
PHP Code:
User-agent: *
Disallow: /forum
Hope this helps.
vinyl-junkie is offline   Reply With Quote
Old 05-14-2004, 04:21 AM   #4
Siava
Green Mole
 
Join Date: May 2004
Location: Russia (Saint-Petersbrg)
Posts: 16
Pulsar-san
No, topic about LOCAL forum installed on my computer in local network

vinyl-junkie
I know, but I need to indexing some files in the forum folder (viewtopic.php, viewforum.php and somthing...).

My robots.txt exclude folders and files that I don't want indexing:

User-agent: *
Disallow: /forum/admin
... other folders ...(I'm skiping other folders)
Disallow: /forum/posting.php
... other files ....

With disallow folders all ok, but this "other files" is not disallow and there are indexing Why??

During spidering I see:

Exclude paths :
- forum/admin
........................(I'm skiping other folders)
- forum/posting\.php
.....................(and other php files with .\)

What a posting.\php ??? Why with .\ ??
Siava is offline   Reply With Quote
Old 05-14-2004, 05:20 AM   #5
Pulsar-san
Green Mole
 
Join Date: May 2004
Location: France
Posts: 8
Oups ! Sorry. I missed the "local"

If it is "\." I'd say that the point is escaped, it is just a missing stripslashes()

Now, why it is indexed, from what I know
/posting.php?mode=newtopic&f=7
and
/posting.php?mode=newtopic&f=5
are considered as 2 different pages by spiders, so
Disallow: /forum/posting.php
will only forbid access to specifically posting.php, without any params.

I'm not sure about that, but this is what I have understood.
I have not seen the possibility to use wildchars in filenaming for robots.txt
Except, perhaps, adding posting.php in the
PHP Code:
// regular expression to ban useless external links in index
define('BANNED','^ad\.|banner|doubleclick'); 

Last edited by Pulsar-san; 05-14-2004 at 05:28 AM.
Pulsar-san is offline   Reply With Quote
Old 05-14-2004, 07:47 AM   #6
Siava
Green Mole
 
Join Date: May 2004
Location: Russia (Saint-Petersbrg)
Posts: 16
Pulsar-san
Quote:
Disallow: /forum/posting.php will only forbid access to specifically posting.php, without any params.


Quote:
Except, perhaps, adding posting.php in the
PHP Code:
define('BANNED','^ad\.|banner|doubleclick|posting.php'); 
Yes, I was add this file into config and make BANNED rule, but this file successfully indexing
__________________
Siava.ru
Siava is offline   Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Local install just not spidering zuzu Troubleshooting 0 02-28-2006 02:53 AM
Local install just not spidering sealless Troubleshooting 4 02-04-2006 07:33 AM
Local MacOSX permissions yoyopoc Script Installation 1 07-18-2005 04:07 PM
local index xorex How-to Forum 3 07-25-2004 05:46 PM
On a local network guy Mod Requests 1 04-20-2004 01:07 PM


All times are GMT -8. The time now is 11:21 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.