Indexing a directory with many files

rafarspd · 12-19-2004, 07:46 AM

This old boy's brain

has stopped functioning - HELP please.

Original version was 1.8.0.
PHPDIG was removed from the PHP area and the SQL data dropped.
I started afresh with V1.8.6.
My problem was only apparent after installation of the latest version.

I have a sub-directory which contains 79 *.txt files (2K to 13K each) which are sequentially numbered. [Newsletters, generated and uploaded monthly].
The search results only display 17 of them and it appears that they are not in number order but in the order stored on the ISP's HDD.

I have changed the following (as an experiment):

define('SPIDER_MAX_LIMIT',50);
define('RESPIDER_LIMIT',50);
define('LINKS_MAX_LIMIT',50);
define('RELINKS_LIMIT',50);
with no difference.

Obviously I am looking in the wrong area so any ideas!

Charter · 12-19-2004, 10:02 AM

Set "search depth" to a large number, set "links per" to zero, set LIMIT_TO_DIRECTORY to false.

rafarspd · 12-21-2004, 02:49 AM

Thanks, it help a bit but I now have to confess to the main reason for the problem.

I forgot a basic principle ?

If you want PHPDIG to find a file then it must have a URL somewhere pointing to it.

I had removed some references to older text files from a page.

Problem is now cured by adding the URL's (with no text) at the bottom of the html just above </body></html>.
They are not seen when viewing but PHPDIG finds and indexes them.

12-19-2004, 07:46 AM	#1
rafarspd Orange Mole Join Date: Nov 2003 Location: High Wycombe, Bucks. UK Posts: 35	Indexing a directory with many files This old boy's brain has stopped functioning - HELP please. Original version was 1.8.0. PHPDIG was removed from the PHP area and the SQL data dropped. I started afresh with V1.8.6. My problem was only apparent after installation of the latest version. I have a sub-directory which contains 79 *.txt files (2K to 13K each) which are sequentially numbered. [Newsletters, generated and uploaded monthly]. The search results only display 17 of them and it appears that they are not in number order but in the order stored on the ISP's HDD. I have changed the following (as an experiment): define('SPIDER_MAX_LIMIT',50); define('RESPIDER_LIMIT',50); define('LINKS_MAX_LIMIT',50); define('RELINKS_LIMIT',50); with no difference. Obviously I am looking in the wrong area so any ideas! __________________ Regards Bob Finch

12-19-2004, 10:02 AM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Set "search depth" to a large number, set "links per" to zero, set LIMIT_TO_DIRECTORY to false. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

12-21-2004, 02:49 AM	#3
rafarspd Orange Mole Join Date: Nov 2003 Location: High Wycombe, Bucks. UK Posts: 35	Thanks, it help a bit but I now have to confess to the main reason for the problem. I forgot a basic principle ? If you want PHPDIG to find a file then it must have a URL somewhere pointing to it. I had removed some references to older text files from a page. Problem is now cured by adding the URL's (with no text) at the bottom of the html just above </body></html>. They are not seen when viewing but PHPDIG finds and indexes them. __________________ Regards Bob Finch

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How to limit indexing to a specific directory?	Beans	How-to Forum	1	08-15-2005 11:03 AM
How to index only local directory files?	sf44	How-to Forum	0	01-28-2005 03:56 AM
Can only index files in a single directory	gcrachel	Troubleshooting	5	09-28-2004 07:23 AM
Not indexing my files	Spider	Troubleshooting	5	07-22-2004 06:12 PM
How to index a directory with pdf files	simonced	How-to Forum	3	02-13-2004 11:41 AM