Bug in grabbing urls from the page

[Konstantine]

Hello again! I found, that if link looks like:

≶a href=next/index.html>next/index.html≶/a>

i.e. without quotes, phpdig don't follow it!

[Konstantine]

Sorry, no bug found

[Charter]

Hi Konstantine, and welcome to PhpDig.net!

Thanks for the contributions too. It's good to have other people review the code and offer input.

[Konstantine]

I had that problem on my work but can't verify it now and tell what really happend.

[Konstantine]

Hi again, so there IS a bug. I can't explain why is it, but you can see it. Try to index http://madboard.ru and look at the page http://madboard.ru/index.html?act=do&code=43. So you will not find links such as http://madboard.ru/index.html?act=do&code=45. Try it. I didn't found why is it happening. Any sugestions?

[Konstantine]

I found it (BUG)!!! And it's not in PHPDIG It's in PHP function

I use PHP Version 4.3.3, OS Linux, so the bug is in parse_url function

You can find it out on site madboard.ru. If you'll try to index it you'll find about 42 pages (if the bug is in your version of PHP).

So change in robot_functions.php in function function phpdigRewriteUrl($eval)

code:

PHP Code:

$url = @parse_url(str_replace('\\\\'"','',$eval));
if (!isset($url['path'])) {
     $url['path'] = '';
} 


by following code:

PHP Code:

$url = @parse_url(str_replace('\\\\'"','',$eval));
$url['query']=str_replace("&","&",$url['query']);
if (!isset($url['path'])) {
     $url['path'] = '';
} 


After that try to index madboard.ru again

You'll find about 400 pages!!!

the bug is:

if you try to parse url http://madboard.ru/index.html?act=do&code=43 you'll get in 'query' line act=do&code=43

Any questions?

If you tried it and it was as I said, please reply on this message

[Charter]

Hi. Speaking of & versus & there is a small bug in version 1.8.0 when PHPDIG_SESSID_REMOVE is set to true. To fix do the following.

In robot_functions.php find:

PHP Code:

$eval = str_replace("&&","&",$eval);
$eval = eregi_replace("[?][&]","?",$eval);
$eval = eregi_replace("&$","",$eval); 


and replace with:

PHP Code:

$eval = str_replace("&&","&",$eval);
$eval = str_replace("?&","?",$eval);
$eval = eregi_replace("&$","",$eval); 


Also, in robot_functions.php find:

PHP Code:

$file = str_replace("&&","&",$file);
$file = eregi_replace("[?][&]","?",$file);
$file = eregi_replace("&$","",$file); 


and replace with:

PHP Code:

$file = str_replace("&&","&",$file);
$file = str_replace("?&","?",$file);
$file = eregi_replace("&$","",$file);