PhpDig.net - View Single Post - Can't get PHPDig to index an htaccess protected site

Charter · 02-23-2004, 12:42 PM

Hi. The type of links that PhpDig follows are those that match the regex in this thread. The comma is already allowed in the query string so I don't think that is the problem but am not exactly sure.

However, you had mentioned crawling secure links. One thing came to mind. The fsockopen error and/or NOFILE would make sense I guess if PhpDig was looking for file at http rather than at https or visa versa.

Try another test using a query string containing 1,1,1,1,1,1,0,0 but have everything using http instead of https, and use links not fopen. Does it index then?

If it does, then in the above linked thread, look at the code in the phpdigIndexFile function and change http to [a-z]{3,5} in the while line.

Also, set PHPDIG_IN_DOMAIN to true in the config file and apply the code change in this thread.

02-23-2004, 12:42 PM	#19
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. The type of links that PhpDig follows are those that match the regex in this thread. The comma is already allowed in the query string so I don't think that is the problem but am not exactly sure. However, you had mentioned crawling secure links. One thing came to mind. The fsockopen error and/or NOFILE would make sense I guess if PhpDig was looking for file at http rather than at https or visa versa. Try another test using a query string containing 1,1,1,1,1,1,0,0 but have everything using http instead of https, and use links not fopen. Does it index then? If it does, then in the above linked thread, look at the code in the phpdigIndexFile function and change http to [a-z]{3,5} in the while line. Also, set PHPDIG_IN_DOMAIN to true in the config file and apply the code change in this thread. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.