|
Download -
PHP Reference -
Links -
Demo 1.8.9 RC1 -
API
License - Credits - Copying - Changelog - Documentation |
|
|
09-07-2003, 12:49 PM
|
#1 |
|
Green Mole
Join Date: Sep 2003
Posts: 3
|
Problem with links
Hi. Just installed phpDig, because I´m searching for a search engine for my website. First I tried to make it dig since the root, and it found many links. However, I only wanted it to search the following files:
http://www.oc-zone.com/modules.php&name=Reviews http://www.oc-zone.com/modules.php&name=Articles http://www.oc-zone.com/modules.php&name=News How can this be done? If I try to do it with these URL´s, I get no new links  Thank you for any help. |
|
|
|
09-08-2003, 04:00 PM
|
#2 |
|
Green Mole
Join Date: Sep 2003
Location: USA
Posts: 1
|
You need to create a robot.txt file in your root directory that excludes all but those directories. There is a free program at download.com that will create a robot file for you. Just search for it.
__________________
James Woodford |
|
|
|
|
09-08-2003, 04:36 PM
|
#3 |
|
Green Mole
Join Date: Sep 2003
Posts: 3
|
Thank you
|
|
|
|
|
09-08-2003, 04:56 PM
|
#4 |
|
Head Mole
Join Date: May 2003
Posts: 2,539
|
Hi. Here are a couple of simple robots.txt examples:
# Disallow PhpDig from dir_to_exclude User-agent: PhpDig Disallow: /dir_to_exclude/ # Disallow all robots from all directories # assuming the robots obey robots.txt User-agent: * Disallow: /
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
|
|
|
 |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Spider problem, Search mb_ereg_replace problem. (Fixed?!) | cpeter | Troubleshooting | 0 | 02-24-2006 02:56 PM |
| the same problem - not index all links | redlock | Troubleshooting | 0 | 12-28-2004 07:36 AM |
| Problem Solved: No links found | calvarym | Troubleshooting | 0 | 11-10-2004 03:24 PM |
| Problem with indexing "links found : 0" | IAMHHawaii | Troubleshooting | 1 | 09-20-2004 01:06 PM |
| problem indexing dynamic links. | orbitalz | Troubleshooting | 3 | 04-30-2004 08:47 PM |
