|
12-30-2004, 07:32 AM | #1 |
Green Mole
Join Date: Dec 2004
Posts: 3
|
Exclude filenames with certain attributes?
All installed and spidering nicely. However, the directory in which the content resides contains a mailarchive of 8000 messages, each of which has its own .html file. Additionally it contains indexes for every 15 messages, again each index has its own .html file.
If the files are in the form: http://www.mydomain.com/message001.html http://www.mydomain.com/message002.html http://www.mydomain.com/message003.html etc and http://www.mydomain.com/index001.html http://www.mydomain.com/index002.html is it possible to restrict the spider so that any file which contains the characters "index" in the title is ignored or, alternatively, restrict the spider such that it only searches files which start with the characters "message"? I had hoped that robots.txt would have been the answer, but you cannot use wildcards to specify exclusions. Typing in a list of the path of every file beginning with "index" is not an option (there are over 150 of them) and in any case, the content is updated every day.... Any help MUCH appreciated |
12-30-2004, 01:29 PM | #2 |
Green Mole
Join Date: Oct 2004
Posts: 27
|
__________________
rAdoN was here |
12-30-2004, 04:00 PM | #3 |
Green Mole
Join Date: Dec 2004
Posts: 3
|
Thanks for your help, which is much appreciated, but I am still failing. Here is what I did:
I commented out the old FORBIDDEN_EXTENSIONS line and replaced it with PHP Code:
Then in robot_functions php file, I changed the 3 instances of regs[5] around the !eregi(BANNED,$regs[5]) piece for regs[2] However, the spider still continues to index files named index123.html Where am I going wrong? Last edited by the_hut2; 12-30-2004 at 04:11 PM. |
12-30-2004, 04:22 PM | #4 |
Green Mole
Join Date: Oct 2004
Posts: 27
|
like this
PHP Code:
remove cgi|php|asp|pl| if you want index of links end with that extensions you need to go admin update - delete links you no want - run the cleans - edit FORBIDDEN_EXTENSIONS - index
__________________
rAdoN was here |
12-30-2004, 04:37 PM | #5 | |
Green Mole
Join Date: Oct 2004
Posts: 27
|
Quote:
__________________
rAdoN was here |
|
12-31-2004, 02:17 AM | #6 |
Green Mole
Join Date: Dec 2004
Posts: 3
|
Great! That works!
Thanks for your help. |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
exclude filenames | felyx | Troubleshooting | 0 | 11-20-2006 09:29 PM |
Get text from alt and title images attributes? | djuritz | How-to Forum | 0 | 07-14-2006 06:05 AM |
New Exclude Option | josegringo | How-to Forum | 2 | 02-17-2005 02:48 PM |
How to index filenames? | mordormx | How-to Forum | 0 | 10-16-2004 06:03 PM |
exclude doesn't really work? | manute | Troubleshooting | 15 | 10-20-2003 02:20 AM |