Indexing a directory with many files
 

This old boy's brain :confused: has stopped functioning - HELP please.

Original version was 1.8.0.
PHPDIG was removed from the PHP area and the SQL data dropped.
I started afresh with V1.8.6.
My problem was only apparent after installation of the latest version.

I have a sub-directory which contains 79 *.txt files (2K to 13K each) which are sequentially numbered. [Newsletters, generated and uploaded monthly].
The search results only display 17 of them and it appears that they are not in number order but in the order stored on the ISP's HDD.

I have changed the following (as an experiment):

define('SPIDER_MAX_LIMIT',50);
define('RESPIDER_LIMIT',50);
define('LINKS_MAX_LIMIT',50);
define('RELINKS_LIMIT',50);
with no difference.

Obviously I am looking in the wrong area so any ideas!

Set "search depth" to a large number, set "links per" to zero, set LIMIT_TO_DIRECTORY to false.

Thanks, it help a bit but I now have to confess to the main reason for the problem.

I forgot a basic principle ? :bang:
If you want PHPDIG to find a file then it must have a URL somewhere pointing to it.

I had removed some references to older text files from a page.

Problem is now cured by adding the URL's (with no text) at the bottom of the html just above </body></html>.
They are not seen when viewing but PHPDIG finds and indexes them.