Crawler speed improvement (although affects limit)

[marco]

I had the problem phpdigExplore() returns to many duplicate links. This caused the spider to check 100s of duplicate URLs, which caused a slowdown, and the 1000 pages limit was hit quite fast.

Finally I added the following code at the end of phpDigExplore():

PHP Code:

if(!$_SESSION["links"]) $_SESSION["links"]=array();
$resultlinks = array();
foreach($links as $link){
    if(!array_search($link, $_SESSION["links"])){
        $_SESSION["links"][]=$link;
        $resultlinks[]=$link;
    }
}
return $resultlinks; 


I don't know whether this modification is useful or harms other components. But for the moment, it works.