Indexing hangs, nothing in db

WunderStump · 01-10-2004, 08:54 AM

I installed PHPdig quite a few months ago and it worked like a charm, indexing my site with hundreds of pages with ease. After a while, I noticed it now doesn't really get anywhere with my main site www.kevinfreitas.net
or any subdomains like lu.kevinfreitas.net or sister.kevinfreitas.net

Here's my phpinfo.

When I run indexing it doesn't show me the indexing in progress like it used to. Rather it just seems to hang on the admin homepage.

I've designed my site to conform to CSS and XHTML standards. This wasn't a problem in the past, like I mentioned, but it might help.

I've also just newly installed version 1.6.5 to start out clean. I haven't changed any settings.

Thanks in advance!

Rolandks · 01-11-2004, 08:08 AM

Hmm, yes it hangs. I also can't index the main site www.kevinfreitas.net ??!

-roland-

WunderStump · 01-11-2004, 03:54 PM

What the heck. Well, that helps me know that my PHPdig is probably just fine. So it's either my server or my site. I have a redirect (header function) to move visitors from my root domain to a journal entry on the site. Could that have something to do with it?

I'll also set PHPdig loose on another site or two hosted on the same server to see how it performs. That'll give me a good idea whether the problem lies with my server or site.

Cheers!

Charter · 01-11-2004, 04:10 PM

Hi. I did some testing to see where the hang occurs. The spider.php file calls phpdigGetSiteFromUrl which calls phpdigDetectDir which calls phpdigTestUrl and I think it is the phpdigTestUrl function that PhpDig hangs on because PhpDig is waiting to hear back from the server but it doesn't look like it gets a response.

WunderStump · 01-11-2004, 04:38 PM

So, it indexed another site on my server just fine. I removed the "header" php function call and ZIP!!! phpDig started cruising through my site con mucho gusto.

So, how can I get this to work with the redirect? I do this because I have a comments feature for my journal and to pull the correct ones up, I use a date passed within the URL. If I allow my homepage to be a journal entry (at www.kevinfreitas.net rather than www.kevinfreitas.net/home.php?date=20040107 most comments would be associated with the base url rather than each individual entry.

Thanks for the advice all!

Charter · 01-11-2004, 09:23 PM

Hi. Perhaps try a meta tag redirect like the below.

PHP Code:


			
<?php

$today = date("Ymd");

echo <<<END

<html>

<head>

<meta http-equiv="refresh" content="0;url=http://www.kevinfreitas.net/home.php?date=$today">

</head>

<body>

</body>

</html>

END;

?>

Charter · 02-25-2004, 11:36 AM

Hi. There is now a mod to follow header location redirects here.

01-10-2004, 08:54 AM	#1
WunderStump Green Mole Join Date: Jan 2004 Location: Tacoma, WA Posts: 4	Indexing hangs, nothing in db I installed PHPdig quite a few months ago and it worked like a charm, indexing my site with hundreds of pages with ease. After a while, I noticed it now doesn't really get anywhere with my main site www.kevinfreitas.net or any subdomains like lu.kevinfreitas.net or sister.kevinfreitas.net Here's my phpinfo. When I run indexing it doesn't show me the indexing in progress like it used to. Rather it just seems to hang on the admin homepage. I've designed my site to conform to CSS and XHTML standards. This wasn't a problem in the past, like I mentioned, but it might help. I've also just newly installed version 1.6.5 to start out clean. I haven't changed any settings. Thanks in advance!

01-11-2004, 04:10 PM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. I did some testing to see where the hang occurs. The spider.php file calls phpdigGetSiteFromUrl which calls phpdigDetectDir which calls phpdigTestUrl and I think it is the phpdigTestUrl function that PhpDig hangs on because PhpDig is waiting to hear back from the server but it doesn't look like it gets a response. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

01-11-2004, 09:23 PM	#6
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Perhaps try a meta tag redirect like the below. PHP Code: `<?php $today = date("Ymd"); echo <<<END <html> <head> <meta http-equiv="refresh" content="0;url=http://www.kevinfreitas.net/home.php?date=$today"> </head> <body> </body> </html> END; ?>` __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

02-25-2004, 11:36 AM	#7
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. There is now a mod to follow header location redirects here. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Apache hangs when starting to scan a website	djuritz	Troubleshooting	3	07-14-2006 06:10 AM
Phpdig hangs when asked to spider any url using 1.83	steviec	Troubleshooting	0	02-15-2006 01:27 AM
spider hangs on indexing pdf (pstotext)	sushie	External Binaries	7	06-15-2005 06:57 AM
PhpDig hangs... please help	bpeovski	Troubleshooting	2	08-24-2004 11:20 AM
Search Hangs	dsstowe	Bug Tracker	2	03-16-2004 10:33 AM

01-11-2004, 08:08 AM	#2
Rolandks Purple Mole Join Date: Sep 2003 Location: Kassel, Germany Posts: 119	Hmm, yes it hangs. I also can't index the main site www.kevinfreitas.net ??! -roland-

01-11-2004, 03:54 PM	#3
WunderStump Green Mole Join Date: Jan 2004 Location: Tacoma, WA Posts: 4	What the heck. Well, that helps me know that my PHPdig is probably just fine. So it's either my server or my site. I have a redirect (header function) to move visitors from my root domain to a journal entry on the site. Could that have something to do with it? I'll also set PHPdig loose on another site or two hosted on the same server to see how it performs. That'll give me a good idea whether the problem lies with my server or site. Cheers!

01-11-2004, 04:38 PM	#5
WunderStump Green Mole Join Date: Jan 2004 Location: Tacoma, WA Posts: 4	So, it indexed another site on my server just fine. I removed the "header" php function call and ZIP!!! phpDig started cruising through my site con mucho gusto. So, how can I get this to work with the redirect? I do this because I have a comments feature for my journal and to pull the correct ones up, I use a date passed within the URL. If I allow my homepage to be a journal entry (at www.kevinfreitas.net rather than www.kevinfreitas.net/home.php?date=20040107 most comments would be associated with the base url rather than each individual entry. Thanks for the advice all!