problem indexing password-protected directories - Page 2

Loewenherz · 11-27-2003, 02:23 PM

Hi Carter,

thanks for your help. Sorry, it's not so easy for me, to post some informations in english :-(

Quote:

Originally posted by Charter

'http://user:pass@www.vdoh.de/';

Sorry, I cannot post the really username and the password here.

Quote:

Originally posted by Charter

What do you get? I get the following when viewing the HTML source:

Code:

User-agent: *
<br>Disallow:<br>

Yes, I get the same.

Quote:

Originally posted by Charter

Also, do the username and password that you are using in the URL match those that are in the sites table for this site? [/b]

This is the part, I don't understand. Is it necessary to do some changes in the database? If this are right, can you tell me the mysql-Code?

Loewenherz · 11-27-2003, 11:44 PM

Okay, back to the beginning:

There's a complete website. Some is for all visitors, but there's a restricted area only for members of this association. The members wishes to have a search engine, it can be found in the member area and indexing all sites of this project.

PhpDig indexing the public sites without problems. But what have I to do for indexing the sites in the member area? Theres a entry in den .htpasswd with name and password only for the search engine, but how I let this know the program?

Loewenherz · 11-28-2003, 12:21 AM

Maybe I found the point in table phpdig_sites :

2 http://www.vdoh.de/docs/verband/ 20031128101633 user passwd 0 0

Charter · 11-28-2003, 03:13 AM

Hi. Thanks for your responses. I don't want you to post the true username and password. Yes, that is the correct row from the sites table. Are the username and password in that row correct? Do you get errors when you try to crawl http://user:passwd@www.vdoh.de/docs/verband/ rather than http://user:passwd@www.vdoh.de/index.php?

Loewenherz · 11-28-2003, 06:02 AM

Quote:

Are the username and password in that row correct? Do you get errors when you try to crawl http://user:passwd@www.vdoh.de/docs/verband/ rather than http://user:passwd@www.vdoh.de/index.php? [/b]

Username an password are correct. And I have no errors more.
BUT:

Code:

Der Spider arbeitet gerade...
SITE : http://www.vdoh.de/
Ausgeschlossene Pfade :
- @NONE@
1:http://www.vdoh.de/docs/verband/
(Zeit : 00:00:00)
Kein Link in der temporäreren Tabelle
Links gefunden : 1
http://www.vdoh.de/docs/verband/
Optimizing tables...
Indizierung abgeschlossen! [Zurück] zum Admin-Interface.

PhpDig is indexing not all the sites in the member area.

Charter · 11-28-2003, 10:08 AM

Hi. Does the http://www.vdoh.de/docs/verband/ page do a JavaScript or META redirect once logged in? If so, try using the full URL in the redirect. If this doesn't work or isn't applicable, can you setup a password protected test area on your site that I could try and index?

Loewenherz · 11-29-2003, 02:27 AM

Hi Carter,

Quote:

Originally posted by Charter
[b]Does the http://www.vdoh.de/docs/verband/ page do a JavaScript or META redirect once logged in?

No.

I can send you password and username for the member area with PN.

Thanks for your help.

Another problem: I installed PhpDig in /inc/search/. But this file is outside the member area. Okay, I copied the PhpDig files to /docs/verband/suche/. PhpDig works on this place, but I cannot come in the administration from PhpDig (the Browser works without answer). Is there anything to change in the database?

Charter · 11-29-2003, 09:22 AM

Hi. Does a username and password box pop up when you try to enter the administration? There should be nothing to change in the database tables.

Loewenherz · 12-01-2003, 11:37 AM

Quote:

Originally posted by Charter
Does a username and password box pop up when you try to enter the administration?

No. Nothing happens. No pop-up, no error, only an endless waiting.

Charter · 12-01-2003, 12:03 PM

Hi. When I go to http://www.vdoh.de/docs/verband/suche/admin/ I get a popup box asking for a username and password. Is this the same link that you go to and get endless waiting?

Loewenherz · 12-01-2003, 11:43 PM

Yes. I try this site with another browser and I see the pop-ups. But I cannot login. Okay, forget it. I'll using PhpDig in the restricted area for searchin and in the other area for administration. I think, this is possible.

The only important problem is indexing the sites in the member area.

Loewenherz · 12-03-2003, 12:10 AM

Loewenherz · 12-03-2003, 12:35 AM

Quote:

And the second host /docs/verband/ with the password-entry in the database is gone

Okay, I change the database entry of the Root-URL and add username and password. Now, PhpDig found 150 Links! Wow!
In the database I found only:
DataBase status
Hosts : 1 Entries
Pages : 69 Entries

???

But it works!!! Yippie.

renehaentjens · 12-05-2003, 01:30 AM

I have the same problem as Alex in the original post, but as there are no funny characters in my URL, I cannot solve it Alex's way.

Username/password are in the DB, I have no meta-refresh stuff (I don't even know what it is), yet, with .htaccess, like Alex, I never get beyond the first page, and I always get:

links found : 1
http://myserver.../subdir/mystart.php
Optimizing tables...
Indexing complete !

Without .htaccess, all generated pages are indexed...

Any fresh ideas on this?

Charter · 12-05-2003, 08:34 AM

Hi. It's not a fresh idea, but the username and password in the database for the site need to match the username and password, as uncrypted, that are in the .htpasswd file. Another option would be to crawl the directory without the .htaccess file and then reinstate the .htaccess file so when users click a protected search result link they are prompted for their username and password. Other than what Alex wants to do, I haven't been able to replicate the other issues listed in this thread.

11-27-2003, 11:44 PM	#17
Loewenherz Green Mole Join Date: Sep 2003 Posts: 14	Okay, back to the beginning: There's a complete website. Some is for all visitors, but there's a restricted area only for members of this association. The members wishes to have a search engine, it can be found in the member area and indexing all sites of this project. PhpDig indexing the public sites without problems. But what have I to do for indexing the sites in the member area? Theres a entry in den .htpasswd with name and password only for the search engine, but how I let this know the program? Last edited by Loewenherz; 11-27-2003 at 11:48 PM.

11-28-2003, 03:13 AM	#19
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Thanks for your responses. I don't want you to post the true username and password. Yes, that is the correct row from the sites table. Are the username and password in that row correct? Do you get errors when you try to crawl http://user:passwd@www.vdoh.de/docs/verband/ rather than http://user:passwd@www.vdoh.de/index.php? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

11-28-2003, 10:08 AM	#21
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Does the http://www.vdoh.de/docs/verband/ page do a JavaScript or META redirect once logged in? If so, try using the full URL in the redirect. If this doesn't work or isn't applicable, can you setup a password protected test area on your site that I could try and index? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

11-29-2003, 09:22 AM	#23
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Does a username and password box pop up when you try to enter the administration? There should be nothing to change in the database tables. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

12-01-2003, 12:03 PM	#25
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. When I go to http://www.vdoh.de/docs/verband/suche/admin/ I get a popup box asking for a username and password. Is this the same link that you go to and get endless waiting? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

11-28-2003, 12:21 AM	#18
Loewenherz Green Mole Join Date: Sep 2003 Posts: 14	Maybe I found the point in table phpdig_sites : 2 http://www.vdoh.de/docs/verband/ 20031128101633 user passwd 0 0

12-01-2003, 11:43 PM	#26
Loewenherz Green Mole Join Date: Sep 2003 Posts: 14	Yes. I try this site with another browser and I see the pop-ups. But I cannot login. Okay, forget it. I'll using PhpDig in the restricted area for searchin and in the other area for administration. I think, this is possible. The only important problem is indexing the sites in the member area.

12-03-2003, 12:10 AM	#27
Loewenherz Green Mole Join Date: Sep 2003 Posts: 14	After a refresh of indexing the Root-Directory, PhpDig seems to have indexing a part of the member area: Level 1... 16:http://www.vdoh.de/inc/files/beitrittsformular.pdf (Zeit : 00:00:03) 17:http://www.vdoh.de/docs/verband/ (Zeit : 00:00:03) 18:http://www.vdoh.de/docs/verband/recht.php (Zeit : 00:00:03) 19:http://www.vdoh.de/docs/verband/suche/search.php (Zeit : 00:00:03) 20:http://www.vdoh.de/docs/verband/ins.php (Zeit : 00:00:03) 21:http://www.vdoh.de/docs/verband/press.php (Zeit : 00:00:03) 22:http://www.vdoh.de/docs/verband/veran.php (Zeit : 00:00:03) 23:http://www.vdoh.de/docs/verband/aktuelles.php (Zeit : 00:00:03) 24:http://www.vdoh.de/docs/verband/grem.php (Zeit : 00:00:03) Kein Link in der temporäreren Tabelle Links gefunden : 24 http://www.vdoh.de/docs/dienstleistungen.php http://www.vdoh.de/docs/geschaeftsstelle.php http://www.vdoh.de/docs/satzung.php http://www.vdoh.de/docs/beitrag.php http://www.vdoh.de/docs/links.php http://www.vdoh.de/inc/html/navi.php http://www.vdoh.de/docs/anfahrt1.php http://www.vdoh.de/docs/kontakt.php http://www.vdoh.de/docs/passw.php http://www.vdoh.de/docs/impressum.php http://www.vdoh.de/docs/anfahrt.php http://www.vdoh.de/docs/mitglied.php http://www.vdoh.de/docs/personalia.php http://www.vdoh.de/docs/ http://www.vdoh.de/ http://www.vdoh.de/inc/files/beitrittsformular.pdf http://www.vdoh.de/docs/verband/ http://www.vdoh.de/docs/verband/recht.php http://www.vdoh.de/docs/verband/suche/search.php http://www.vdoh.de/docs/verband/ins.php http://www.vdoh.de/docs/verband/press.php http://www.vdoh.de/docs/verband/veran.php http://www.vdoh.de/docs/verband/aktuelles.php http://www.vdoh.de/docs/verband/grem.php Optimizing tables... But I don't found this sites in the index. And the member area has many more sites based on a PHP-News-System. And the second host /docs/verband/ with the password-entry in the database is gone Last edited by Loewenherz; 12-03-2003 at 12:13 AM.

12-05-2003, 01:30 AM	#29
renehaentjens Orange Mole Join Date: Nov 2003 Posts: 69	I have the same problem as Alex in the original post, but as there are no funny characters in my URL, I cannot solve it Alex's way. Username/password are in the DB, I have no meta-refresh stuff (I don't even know what it is), yet, with .htaccess, like Alex, I never get beyond the first page, and I always get: links found : 1 http://myserver.../subdir/mystart.php Optimizing tables... Indexing complete ! Without .htaccess, all generated pages are indexed... Any fresh ideas on this? __________________ René Haentjens, Ghent University

12-05-2003, 08:34 AM	#30
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. It's not a fresh idea, but the username and password in the database for the site need to match the username and password, as uncrypted, that are in the .htpasswd file. Another option would be to crawl the directory without the .htaccess file and then reinstate the .htaccess file so when users click a protected search result link they are prompted for their username and password. Other than what Alex wants to do, I haven't been able to replicate the other issues listed in this thread. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Indexing Password Protected pages (using session variables)	apetersen	How-to Forum	1	03-27-2007 04:18 AM
Hiding Password Protected Pages	joannelee	How-to Forum	2	03-15-2005 10:07 AM
Highlight password-protected content with padlock?	mr_ruskin	How-to Forum	1	12-15-2004 03:59 AM
For Those With Password Protected Sites	bbenson	How-to Forum	3	09-16-2004 04:25 PM
install password protected?	mistafeesh	Script Installation	3	11-10-2003 11:54 PM