For Those With Password Protected Sites

[bbenson]

I've been struggling with phpdig for quite some time, thinking I MUST be doing something wrong.

Some history: I made a website, which is restricted through apache's auth mechanism in httpd.conf, for friends and family some years ago. Last year, I decided I needed a search engine. After fiddling around with a few that didn't work so well, I came across phpdig. I loved it, but couldn't for the life of me get it to work. So, over the last year, when I had time and wasn't too busy with work and family, I would come back to trying to get it to work.

FINALLY, after scouring posts today something clicked and I thought I would share in case anyone was as stupid as I have been... possibly preventing them -- too late for me -- from going bald with frustration!

What I found out was that phpdig was happily going along and not complaining when it couldn't get to the password-protected content of my site. It would just skip over it and LOOK like it had indexed it. Only, when I went to search things nothing was there!! I couldn't figure this out and it was driving me nuts!!

Finally, a post I saw today made me look in the access logs for apache and, lo and behold, I noticed that every time I ran the spider, I was getting 401 errors in the log for the content that was password protected.

To solve, I temporarily removed the password protection from the server, reloaded the httpd.conf, and let phpdig go to work. NOW it indexed everything and I'm happy as a clam.

I hope this helps someone else someday.

[vinyl-junkie]

Hi, bbenson. Welcome to the forums!

That's certainly one way of getting around the authentication problem. However, your site might be vulnerable to unauthorized activity while you're indexing like this.

Have you seen this thread? The processes described there would provide the means to index your site, while keeping phpdig security in place.

Hope this helps.

[bbenson]

Hi Vinyl-Junkie. Thanks for the welcome.

Thanks for the info. My site, however, is a personal site that is pretty small -- and not widely available: no DNS, just hanging off my cable modem . My removing the security while I index isn't a big deal... as I have it set, it only takes about 6 minutes to index what I want of the site.

I am very impressed with this software and many thanks to the forums, I have gotten many answers here over the last year or so... I'm just shy about posting and showing my ignorance.

Thanks again!!

[vinyl-junkie]

Originally Posted by bbenson

I have gotten many answers here over the last year or so... I'm just shy about posting and showing my ignorance.

Please don't ever be shy about asking a question here. If you haven't been able to find an answer to a question in the archives, it just might be something that not only you but a few other people would like to know the answer to.