|
09-15-2004, 01:28 PM | #1 |
Green Mole
Join Date: Dec 2003
Location: Charlotte, NC
Posts: 2
|
For Those With Password Protected Sites
I've been struggling with phpdig for quite some time, thinking I MUST be doing something wrong.
Some history: I made a website, which is restricted through apache's auth mechanism in httpd.conf, for friends and family some years ago. Last year, I decided I needed a search engine. After fiddling around with a few that didn't work so well, I came across phpdig. I loved it, but couldn't for the life of me get it to work. So, over the last year, when I had time and wasn't too busy with work and family, I would come back to trying to get it to work. FINALLY, after scouring posts today something clicked and I thought I would share in case anyone was as stupid as I have been... possibly preventing them -- too late for me -- from going bald with frustration! What I found out was that phpdig was happily going along and not complaining when it couldn't get to the password-protected content of my site. It would just skip over it and LOOK like it had indexed it. Only, when I went to search things nothing was there!! I couldn't figure this out and it was driving me nuts!! Finally, a post I saw today made me look in the access logs for apache and, lo and behold, I noticed that every time I ran the spider, I was getting 401 errors in the log for the content that was password protected. To solve, I temporarily removed the password protection from the server, reloaded the httpd.conf, and let phpdig go to work. NOW it indexed everything and I'm happy as a clam. I hope this helps someone else someday. |
09-15-2004, 05:33 PM | #2 |
Purple Mole
Join Date: Jan 2004
Posts: 694
|
Hi, bbenson. Welcome to the forums!
That's certainly one way of getting around the authentication problem. However, your site might be vulnerable to unauthorized activity while you're indexing like this. Have you seen this thread? The processes described there would provide the means to index your site, while keeping phpdig security in place. Hope this helps. |
09-16-2004, 07:03 AM | #3 |
Green Mole
Join Date: Dec 2003
Location: Charlotte, NC
Posts: 2
|
Hi Vinyl-Junkie. Thanks for the welcome.
Thanks for the info. My site, however, is a personal site that is pretty small -- and not widely available: no DNS, just hanging off my cable modem . My removing the security while I index isn't a big deal... as I have it set, it only takes about 6 minutes to index what I want of the site. I am very impressed with this software and many thanks to the forums, I have gotten many answers here over the last year or so... I'm just shy about posting and showing my ignorance. Thanks again!! Last edited by bbenson; 09-16-2004 at 07:06 AM. |
09-16-2004, 04:25 PM | #4 | |
Purple Mole
Join Date: Jan 2004
Posts: 694
|
Quote:
|
|
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Hiding Password Protected Pages | joannelee | How-to Forum | 2 | 03-15-2005 10:07 AM |
Highlight password-protected content with padlock? | mr_ruskin | How-to Forum | 1 | 12-15-2004 03:59 AM |
admin password-protected - cron job? | manute | Troubleshooting | 3 | 05-15-2004 11:35 AM |
problem indexing password-protected directories | alexp | Troubleshooting | 29 | 12-05-2003 08:34 AM |
install password protected? | mistafeesh | Script Installation | 3 | 11-10-2003 11:54 PM |