Spidering stops after 6 level 1 pages

[Steve]

Hi

Having got this this brilliant search engine to nearly work, I'm having trouble getting it to go all the way...

No matter which page I start the spider on, it appears to always stop after it's spidered 6 level 1 pages. Site remains 'locked' (easily unlocked) and I'm clearing out the database before each attempt, just to make sure, but the problem seems persistent.

I've read all the other relevant posts that I can find, but can't see an answer to this problem (though I don't think I'm the only user to see it).

I've made only one change in config.php - set limit days to 0.

Is there a definitive answer to this problem - something basic in the setup maybe?

Thanks in advance for helping me out ....

Steve

[vinyl-junkie]

Hi, Steve, and welcome to the forum!

I assume you've made sure the folder permissions are set correctly. What site are you trying to spider? And how are you trying to do it? Through the admin panel, or a cron job? Are you getting some sort of error, or is the spider just hanging? Answers to these questions might give us a start on how to help you.

[Steve]

Hi vinyl-junkie

Just to keep you updated......

I set up phpdig 1.8.0 on my own server and tried to spider the remote site and it worked perfectly - no spider hang ups, so this is obviously a clue! Config was exactly the same. I guess this is something to do with my ISP server - maybe it's got some sort of time limit on running php scripts (?). Anyway, I guess it's not your problem, since I've proved that the spidering works properly - it's just a pain in the arse when when it bombs out on my ISP's server. Can I solve this with a cron job? (I've heard of these somewhere, but I don't have a clue what they are!

Thanks for a really great software package!

Steve

[vinyl-junkie]

Originally posted by Steve
Can I solve this with a cron job? (I've heard of these somewhere, but I don't have a clue what they are!

Glad to hear you were able to spider your site. We like success stories around here.

There is an excellent tutorial on cron jobs here. Basically it's like a batch job that runs on whatever dates and times you specify, so you can spider your site automatically.

Hope this helps.