only indexes the first page...

majestique · 04-08-2004, 04:12 PM

when i run "dig this!" it only indexes the index.html page.. it doesn't even go to the links w/in my index page.. anyone know why this is happening? thanks a lot! =)

vinyl-junkie · 04-08-2004, 05:56 PM

What search depth did you select? If you left it at the default, that's probably the problem. Try changing the search depth to 5 and see what happens.

Hope this helps.

BTW, welcome to the forums, majestique. We're glad to have you here.

majestique · 04-08-2004, 06:48 PM

thanks for the welcome

this script is the best search script .. and i've definitely looked ALL over.. anyway.. i tried setting it to 4 and 5.. and it would still do give me only 1 page.. so what i did was enter in a sub folder within the site.. and that worked beautifully.. the weird thing was.. it indexed my wmv files!? what the heck.. anyway i just went ahead and deleted that.. and entered in the videos folder in my dissallow in robots.txt..

also when i clicked on dig this.. the page seemingly halted.. but it isn't because when i close it.. and enter in the admin area.. i notice my #s going up as I refresh.. shouldn't the temporary table be 0? it's currently stuck at
"Temporary table : 79 Entries "

here's what my database status looks like:
Hosts : 1 Entries
Pages : 36 Entries
Index : 4217 Entries
Keywords : 145076 Entries
Temporary table : 79 Entries

vinyl-junkie · 04-08-2004, 07:06 PM

The temporary table should be empty when you have finished with the spidering process. Sometimes the process hangs for no good reason, and I have yet to figure out why. I've had that happen a couple of times, but when I started over, it worked like a charm. Go figure.

You might try emptying the phpDig tables and starting over. Before you re-index though, make sure the value for LIMIT_DAYS is set to zero so you'll be able to do that.

If you still have problems, let us know. We'll be glad to help.

majestique · 04-08-2004, 08:02 PM

thanks for the fast reply Pat =) I did exactly what u said.. here's what i got.. strange problem huh? i've setup this search engine before.. no problem.. odd..

--------------------------------------------------------------------------------
SITE : http://www.mysite.com/
Exclude paths :
- video
1:http://www.mysite.com/
(time : 00:00:06)
No link in temporary table

--------------------------------------------------------------------------------

links found : 1
http://www.mysite.com/
Optimizing tables...
Indexing complete !
--------------------------------------------------------------------------------
[Back] to admin interface.

vinyl-junkie · 04-08-2004, 08:18 PM

Hmmm. Sounds like you have something in your robots.txt file that is blocking everything. Either that or did you use:

Code:

<!-- phpdigExclude -->
<!-- phpdigInclude -->

comments to exclude any links? That's another thing that could potentially trip you up.

majestique · 04-08-2004, 08:22 PM

yeah u were exactly right =) i fixed the robot.. and i'm back to that old problem again

majestique · 04-08-2004, 08:29 PM

i think i got it now.. i did everything from scratch... now it seems to work =) thanks Pat!!!

vinyl-junkie · 04-08-2004, 08:34 PM

Kewl! Glad you got it working!

04-08-2004, 04:12 PM	#1
majestique Green Mole Join Date: Apr 2004 Posts: 16	only indexes the first page... when i run "dig this!" it only indexes the index.html page.. it doesn't even go to the links w/in my index page.. anyone know why this is happening? thanks a lot! =)

04-08-2004, 06:48 PM	#3
majestique Green Mole Join Date: Apr 2004 Posts: 16	thanks for the welcome this script is the best search script .. and i've definitely looked ALL over.. anyway.. i tried setting it to 4 and 5.. and it would still do give me only 1 page.. so what i did was enter in a sub folder within the site.. and that worked beautifully.. the weird thing was.. it indexed my wmv files!? what the heck.. anyway i just went ahead and deleted that.. and entered in the videos folder in my dissallow in robots.txt.. also when i clicked on dig this.. the page seemingly halted.. but it isn't because when i close it.. and enter in the admin area.. i notice my #s going up as I refresh.. shouldn't the temporary table be 0? it's currently stuck at "Temporary table : 79 Entries " here's what my database status looks like: Hosts : 1 Entries Pages : 36 Entries Index : 4217 Entries Keywords : 145076 Entries Temporary table : 79 Entries Last edited by majestique; 04-08-2004 at 06:50 PM.

04-08-2004, 08:02 PM	#5
majestique Green Mole Join Date: Apr 2004 Posts: 16	thanks for the fast reply Pat =) I did exactly what u said.. here's what i got.. strange problem huh? i've setup this search engine before.. no problem.. odd.. -------------------------------------------------------------------------------- SITE : http://www.mysite.com/ Exclude paths : - video 1:http://www.mysite.com/ (time : 00:00:06) No link in temporary table -------------------------------------------------------------------------------- links found : 1 http://www.mysite.com/ Optimizing tables... Indexing complete ! -------------------------------------------------------------------------------- [Back] to admin interface. Last edited by majestique; 04-08-2004 at 08:15 PM.

04-08-2004, 08:18 PM	#6
vinyl-junkie Purple Mole Join Date: Jan 2004 Posts: 694	Hmmm. Sounds like you have something in your robots.txt file that is blocking everything. Either that or did you use: Code: <!-- phpdigExclude --> <!-- phpdigInclude --> comments to exclude any links? That's another thing that could potentially trip you up.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
New phpDig indexes whole site now or?	JAB Creations	The Mole Hole	0	11-07-2007 04:14 AM
Spider Indexes an Error Page	alisan	Troubleshooting	2	02-13-2006 12:14 PM
db indexes	baskamer	Script Installation	1	12-17-2004 11:07 AM
how to exchange indexes between phpdig instances?	leonardburton	How-to Forum	3	12-05-2004 08:15 AM
old indexes	baskamer	How-to Forum	1	03-31-2004 08:30 AM

04-08-2004, 05:56 PM	#2
vinyl-junkie Purple Mole Join Date: Jan 2004 Posts: 694	What search depth did you select? If you left it at the default, that's probably the problem. Try changing the search depth to 5 and see what happens. Hope this helps. BTW, welcome to the forums, majestique. We're glad to have you here.

04-08-2004, 07:06 PM	#4
vinyl-junkie Purple Mole Join Date: Jan 2004 Posts: 694	The temporary table should be empty when you have finished with the spidering process. Sometimes the process hangs for no good reason, and I have yet to figure out why. I've had that happen a couple of times, but when I started over, it worked like a charm. Go figure. You might try emptying the phpDig tables and starting over. Before you re-index though, make sure the value for LIMIT_DAYS is set to zero so you'll be able to do that. If you still have problems, let us know. We'll be glad to help.

04-08-2004, 08:22 PM	#7
majestique Green Mole Join Date: Apr 2004 Posts: 16	yeah u were exactly right =) i fixed the robot.. and i'm back to that old problem again

04-08-2004, 08:29 PM	#8
majestique Green Mole Join Date: Apr 2004 Posts: 16	i think i got it now.. i did everything from scratch... now it seems to work =) thanks Pat!!!

04-08-2004, 08:34 PM	#9
vinyl-junkie Purple Mole Join Date: Jan 2004 Posts: 694	Kewl! Glad you got it working!