PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 04-06-2004, 01:52 PM   #1
bigals
Orange Mole
 
Join Date: Nov 2003
Posts: 41
Not Spidering Deep enough (strange site i have tho)

Hi i went over this problem in the thread I psted below, after I managed to get my site to spider my php files, i came accroos the problem of not spidering deep enough:

Old Thread:
http://www.phpdig.net/showthread.php...5&pagenumber=2

It spiders 20 deep, and thats as far as it goes, but I have folders that contain over 60 php files, and each one needs to be spidered, so can i set the spider depth to about 70?

so for these big folders, 20 of the php fiiles are spidered and the rest are not...

can someone explain how to dig a little deeper

cheers,

Alex.
bigals is offline   Reply With Quote
Old 04-06-2004, 07:17 PM   #2
vinyl-junkie
Purple Mole
 
Join Date: Jan 2004
Posts: 694
This is just a guess, but have you tried changing this code in config.php:
Code:
define('SPIDER_MAX_LIMIT',20);
to this?
Code:
define('SPIDER_MAX_LIMIT',70);
vinyl-junkie is offline   Reply With Quote
Old 04-07-2004, 12:46 AM   #3
bigals
Orange Mole
 
Join Date: Nov 2003
Posts: 41
O right I see, i'll try that and see how it goes!

cheers!
bigals is offline   Reply With Quote
Old 04-07-2004, 03:29 AM   #4
bigals
Orange Mole
 
Join Date: Nov 2003
Posts: 41
DAM its not working!

Its still spidering only 20 results, i think its because my php page are only generating 20 results for the page, when it reaches 20 results it creates a button that links to the next 20, the button will refresh the page and list results 20 - 40,

you see all the links that are sopidered are created by php and don't exist anywhere else, is that why its not spidering anymore than 20 do you think?

it sounds like its right.

bigals is offline   Reply With Quote
Old 04-10-2004, 11:32 AM   #5
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. PhpDig follows links. If there are no links to the other pages, then PhpDig won't find them.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 04-10-2004, 12:48 PM   #6
bigals
Orange Mole
 
Join Date: Nov 2003
Posts: 41
Yeah cool!

I did what we said before, i've made 1 php page that generates a huge list of links to every index page and it works but it seems to miss some of them,

i only spidered 1 deep because every link was in the same documnet, perhaps if a spider a little deeper it might find the ones its missing out?

alex.
bigals is offline   Reply With Quote
Old 09-19-2004, 06:49 AM   #7
asanad
Green Mole
 
Join Date: Sep 2004
Posts: 11
Unhappy Not crawling deep enough

Hi all

I am trying to crawl my website, for some reason PHPDig is only crawling very partially my website, it does not go deaper than the homepage.

Best Regards:
Amjad Sanad
asanad is offline   Reply With Quote
Old 09-19-2004, 09:07 AM   #8
vinyl-junkie
Purple Mole
 
Join Date: Jan 2004
Posts: 694
What search depth and links per values did you choose? If you left them both at the default of zero, you'll only get the home page crawled. Try changing the search depth to 5, for example, and see what happens. Depending on how many pages phpdig spiders you'll have to set that value even higher.

Hope this helps.
vinyl-junkie is offline   Reply With Quote
Old 09-19-2004, 11:32 PM   #9
asanad
Green Mole
 
Join Date: Sep 2004
Posts: 11
Unhappy Still not spidering deep enough

I set the depth to 20 and links to 0, but, still it is not spidering deep enough
only certain documents in the homepage.
asanad is offline   Reply With Quote
Old 09-21-2004, 06:48 PM   #10
vinyl-junkie
Purple Mole
 
Join Date: Jan 2004
Posts: 694
Find the following in config.php:
Code:
define('SPIDER_MAX_LIMIT',20);
and change it to this:
Code:
define('SPIDER_MAX_LIMIT',40);
Re-spider and see what you get.
vinyl-junkie is offline   Reply With Quote
Reply


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
index just homepage with a cronjob (the mole is too deep) propain How-to Forum 0 02-14-2005 12:02 AM
Deep trouble with spidering Dave A Troubleshooting 2 11-10-2004 12:27 AM
version 1.8.2, 1.8.3 doesn't fully spidering the site Siava Troubleshooting 15 07-19-2004 01:55 AM
Spidering issue with my site pager Troubleshooting 5 01-19-2004 10:05 AM
Strange indexing problem on my site drbill Troubleshooting 9 01-01-2004 01:29 PM


All times are GMT -8. The time now is 07:35 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.