Spidering returns very few keyword results

[e_talis]

Hi all,
I am using phpDig 1.6.1 on my site. At first when I installed it it did a good spider and returned many keywords. Later when I deleted some link entries and tried to respider the site, the number of keywords became very few and now when someone does a search most words return "No Result" but the words were returning many results before I went on the deleting spree. I have tried to clear the whole database to see if I can remove any Logs but it stiill gives few or no results. What can I do to get back my many keywords?

[bloodjelly]

Hi e_talis -

First, it couldn't hurt to upgrade to 1.8.3 before you get started again, as it offers lots of new features and a few bug fixes.

As for increasing the number of keywords found, try 1) setting the search depth higher, and 2) making sure all your pages are linked to on your site so that phpDig can find them. You might want to set up a site map with links to the main parts of your site. Just make sure phpDig can find a link to your site map. Good luck.

[e_talis]

Hi bloodjelly,

Okay, I will upgrade by the end of the day and see how 1.8.3 works. I am using a search depth of 5, which was working quite fine when I first installed the mole. I will try and test it with 10 and see how things work out. About the site map, the spidering is detecting all the pages+links in my site but it is not picking up enough keywords from the pages it finds (it was in the begining before I started deleting links).
Thanks.

[e_talis]

bloodyjelly,
I have installed 1.8.3 as you had suggested...
I still get the same proble and it is spidering at a very slow rate. Any suggestions on how to boost it up?

[bloodjelly]

Hmm...as for the spidering being slow, check out this thread: http://www.phpdig.net/showthread.php?s=&threadid=1066

Why it's not gathering as many keywords I'm not sure. The new version is supposed to get them differently from what I understand, but it should still get a good number. Can you post a URL where we can check it out?

[e_talis]

I have changed to

PHP Code:

sleep(1); 


in the spider.php and the spidering is a bit faster now.
Is there a way of excluding php errors in the summary text? ... maybe I should silence the functions that give the errors eg.

PHP Code:

@setcookie("....") 


[bloodjelly]

You could silence the errors, or set a lower level of error reporting (http://www.php.net/error_reporting), but why are you getting php errors in the first place?

[vinyl-junkie]

To suppress 404 errors, make sure you have this set like so in config.php:

PHP Code:

define('SILENCE_404S',true); 


Note: This is a new feature of 1.8.3.

[e_talis]

Hello,
I have silenced the 404s, that's ok. It seems while the mole is spidering, it outputs something therefore when it reaches a page with setcookie or session_start, a header cannot be sent becoz of the output from phpdig. The same pages are working fine when I browse them.