PhpDig only with Mysql

[vodevil]

I want to make Phpdig work only with Mysql because the folder text_content take much disk space on my server (100MB) , so I want to know if somebody have already made this ??????


excuse me my english is not perfect

[bloodjelly]

Making PhpDig store the content in SQL tables won't take up any less space (I guess unless you used compression). You can limit the storage of text content by changing this line in the config file:

define('CONTENT_TEXT',0);

This way no text files will be created. Otherwise I guess you'd have to make some fulltext tables, then go through the code, and everywhere you see an fopen (in phpdigWriteText (robot_functions.php) for example), change it to an SQL query.

[vodevil]

Ok bloodjelly at this time a test this solution , if it work i'll submit my modified phpDig version to the community !

[vodevil]

I have modifieed this function :

//=================================================
//store a content_text from a spider_id
function phpdigWriteText($relative_script_path,$spider_id,$text,$ftp_id='') {
global $br;
if (CONTENT_TEXT == 1) {

$file_text_path = $relative_script_path.'/'.TEXT_CONTENT_PATH.$spider_id.'.txt';
if ($f_handler = @fopen($file_text_path,'w')) {
reset($text);
while (list($n_chunk,$text_to_store) = each($text)) {
fputs($f_handler,wordwrap($text_to_store));
}
fclose($f_handler);
@chmod($file_text_path,0666);
//here the ftp case
if (FTP_ENABLE) {
$ftp_id = phpdigFtpKeepAlive($ftp_id);
@ftp_delete($ftp_id,$spider_id.'.txt');
$res_ftp = false;
$try_count = 0;
while (!$res_ftp && $try_count++ < 10) {
$res_ftp = @ftp_put($ftp_id,$spider_id.'.txt',$file_text_path,FTP_ASCII);
if (!$res_ftp) {
sleep(2);
}
}
if (!$res_ftp) {
print "Ftp_put error !".$br;
}

}
}
else {
print "Warning : Unable to create the content file $file_text_path ! $br";
}
}
return $ftp_id;
}

By this :

//=================================================
//store a content_text from a spider_id
function phpdigWriteText($relative_script_path,$spider_id,$text,$ftp_id='') {
global $br;
if (CONTENT_TEXT == 1) {


while (list($n_chunk,$text_to_store) = each($text)) {
fputs($f_handler,wordwrap($text_to_store));
}

$requete = "INSERT INTO text_content VALUES ('$spider_id','$text_to_store')";


$query = mysql_query($requete) or die(mysql_error);

}


}

This modification no cause error but phpdig doesn't store information into Mysql

[AllKnightAccess]

I would also like a way to store the text into MySQL, instead of the server. Why is it stored on the server in the first place? Wouldn't the content be better placed in the database?

[Charter]

You don't necessarily have to modify the PhpDig 1.8.7 code to store text in MySQL. Try setting the following in the config file instead:

define('SUMMARY_DISPLAY_LENGTH',150);            //Max chars displayed in summary

define('DISPLAY_SNIPPETS',false);                 //Display text snippets

define('DISPLAY_SUMMARY',true);                 //Display description

define('CONTENT_TEXT',0);                    //Activates/deactivates the
                                             //storage of text content.

In search_function.php of PhpDig 1.8.7, here is some relevant code:

             if (DISPLAY_SUMMARY) {
                 $table_results[$n]['text'] = htmlspecialchars(phpdigHighlight($reg_strings,preg_replace("/([ ]{2,}|\n|\r|\r\n)/"," ",ereg_replace('(@@@.*)','',wordwrap($text, SUMMARY_DISPLAY_LENGTH, '@@@')))),ENT_QUOTES);
                 $table_results[$n]['text'] = phpdigSpanReplace($table_results[$n]['text']);
             }

In robot_functions.php of PhpDig 1.8.7, here is some relevant code:

$db_some_text = preg_replace("/([ ]{2,}|\n|\r|\r\n)/"," ",implode("",$text));
if (strlen($db_some_text) > SUMMARY_DISPLAY_LENGTH) {
  $db_some_text = substr($db_some_text,0,SUMMARY_DISPLAY_LENGTH)."...";
}

$first_words = preg_replace("/([ ]{2,}|\n|\r|\r\n)/"," ",$titre_resume)."\n".preg_replace("/([ ]{2,}|\n|\r|\r\n)/"," ",$page_desc['content'].$db_some_text)."...";

Note that up to SUMMARY_DISPLAY_LENGTH characters will get displayed in the search results, and that the highlighting of keywords works differently than when storing text in files, so make sure to try a demo before going live.

This was all changed/improved in PhpDig 1.8.8 RC1 so that highlighting can work the same whether or not you store text in files, and PhpDig 1.8.8 RC1 has a TEXT_STORAGE_AMOUNT option so that you can set how much text to store.

If using PhpDig 1.8.8 RC1, you'll probably want the following config options set to store text in MySQL but not files:

define('DISPLAY_SNIPPETS',true);                 // display text snippets

define('DISPLAY_SUMMARY',false);                 // display description

define('SNIPPET_DISPLAY_LENGTH',150);            // max chars displayed in each snippet

define('TEXT_STORAGE_AMOUNT',10000);             // max characters per page to store in files/tables

define('CONTENT_TEXT',0);                        // activates/deactivates the storage of text content in files

[vodevil]

My solution don't work , it's the good way but phpdig storing nothing into Mysql .

[vodevil]

This is my new modification

//=================================================
//delete a spider reccord and content file
function PhpDigDelSpiderRow($id_connect,$spider_id,$ftp_id='')
{
global $relative_script_path,$ftp_id;
$query = "DELETE FROM ".PhpDig_DB_PREFIX."engine WHERE spider_id=$spider_id";
$result_id = mysql_query($query,$id_connect);
$query = "DELETE FROM ".PhpDig_DB_PREFIX."spider WHERE spider_id=$spider_id;";
$result_id = mysql_query($query,$id_connect);
// PhpDigDelText($relative_script_path,$spider_id,$ftp_id);
   PhpDigDelText($spider_id,$ftp_id);
}



// =============================
// Ecriture dans Mysql
// print "Erreur d'écriture dans Mysql ! $br";

function PhpDigWriteText($relative_script_path,$spider_id,$text,$ftp_id='') {
global $br;
if (CONTENT_TEXT ==1) {

$file_text_path = mysql(" SELECT * FROM 'PhpDigtext_content' ORDER BY 'spider_id' ");

if($f_handler = $file_text_path ) {
reset($text);
while (list($n_chunk,$text_to_store) = each($text)) {

mysql_query(" INSERT INTO PhpDigtext_content (spider_id, content) VALUES ('$spider_id','$text_to_store')", $id_connect);
  }

}

// Here the mysql case
if (FTP_ENABLE) {
$ftp_id = PhpDigFtpKeepAlive($ftp_id);

// @delete @ftp_delete($ftp_id,$spider_id.'.txt');

$delete = mysql_query(" DELETE 'PhpDigtext_content' WHERE 'spider_id' = '$spider_id' ");
$res_ftp = false;
$try_count = 0;
while (!$res_ftp && $try_count++ < 10) {

  $res_ftp = mysql(" INSERT INTO PhpDigtext_content (spider_id) VALUES ('$spider_id'), $id_connect);
                if (!$res_ftp) {
                            sleep(2);

}

                  if (!$res_ftp) {

                                   echo 'Unable to wgite in ! $br';

         }
 
      }

    }

       else {
              echo 'Warning : Imposible d\'écrire dans Mysql file $file_text_path ! $br';

 }

}

 return $ftp_if;

}

//=================================================
//delete a content_text from a spider_id
function PhpDigDelText($relative_script_path,$spider_id,$ftp_id='')
{
if (CONTENT_TEXT == 1)
{

//there delete the ftp file
if (FTP_ENABLE && $ftp_id)
             @$delete = 1;
}
}



// =============================

function PhpDigFtpConnect()
{
 if (CONTENT_TEXT == 1 && FTP_ENABLE== 1) {
 $count = 0;
 global $br;

 while ($count++ < 10) {

                $ftp_id = 1;


    }
    
    return $ftp_id; 
 
  }
   sleep(2); 
 
}


// =============================
// close ftp if exis

 function PhpDigFtpClose($ftp_id)
{
 if ($ftp_id)
    @ftp_quit($ftp_id);
}

It's work at 98% but this function , cause a bug ,

//=================================================
//delete a content_text from a spider_id
function phpdigDelText($relative_script_path,$spider_id,$ftp_id='')
{
if (CONTENT_TEXT == 1)
{


$query = "SELECT * FROM 'phpdigtext_content' WHERE spider_id = '$spider_id' ";

$file_text_path =mysql_query($query,$id_connect);


//there delete the ftp file


if (FTP_ENABLE && $ftp_id)
    $delete = mysql_query(" DELETE FROM 'phpdigtext_content' WHERE spider_id=$spider_id ");
}

}

[vodevil]

//=================================================
//delete a content_text from a spider_id
function xunDelText($spider_id,$ftp_id='')
{
if (CONTENT_TEXT == 1)
{

if (FTP_ENABLE && $ftp_id)
    $delete = mysql_query(" DELETE * FROM 'xuntext_content' WHERE spider_id=$spider_id ");


   }

}

Don't work