I'm currently facing the same problem (indexing password protected files that don't use .htaccess protection) and found this thread very helpful.
I would like to point out to people that forging a user agent header is very easy, especially with browsers such as opera. If you are going to use the user agent as an authentication method you should edit spider.php and set the user agent to something else and then test for that.
Look for the following lines in admin/spider.php and change to something a little harder to guess:
// set the User-Agent for the file() function
@ini_set('user_agent','PhpDig/'.PHPDIG_VERSION.' (+
http://www.phpdig.net/robot.php)');