|
Download -
PHP Reference -
Links -
Demo 1.8.9 RC1 -
API
License - Credits - Copying - Changelog - Documentation |
|
|
03-29-2004, 10:37 PM
|
#1 |
|
Orange Mole
Join Date: Mar 2004
Posts: 48
|
Admin approval for spider to index external URLs
Letting the spider run wild does a niche search engine more harm than good. As it indexes sites that are not relevant, it uses server resources unnecessarily and causes search results to slow down due to additional load.
A modification I'd really like to see is to have the spider place all external links it finds while indexing a site into a table for later admin approval to index. Upon approval, these urls could be moved to a text file for indexing. |
|
|
 |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| I: PHPDIG can not index 2+ URLs.. ? | PL_90 | Script Installation | 0 | 10-22-2007 08:51 AM |
| Restart spider and index urls in temptable | jerrywin5 | Troubleshooting | 1 | 04-06-2005 02:18 PM |
| Spider External links to a depth of 1 (1.8.3) | kenazo | How-to Forum | 0 | 10-20-2004 07:28 AM |
| phpdig seems to guess some urls and spider it | manute | Troubleshooting | 7 | 04-29-2004 02:49 AM |
| viewing admin/index.php... | ronyotz | Troubleshooting | 3 | 03-24-2004 05:33 AM |
