Remove Index pages

bigals · 03-10-2004, 01:42 AM

Hi I have php dig set up on a web site and I'm having trouble getting round a problem.

PHP DIG indexes pages when spidering, these pages are roots of folders, they appear in the spider directory as '-' entries, these are then listed when a user searches, when they are clicked they just open folder pages on my web site such as:

www.domain.com/html/one/

thats a bit confusing so try this description:

if you have a page called:

www.domain.com/html/one/one.html

the spider indexes this link but also indexes the root folder of that html page, so you get both of these results:

www.domain.com/html/one/one.html
www.domain.com/html/one/

I don't want these folders indexed, i just want the html pages indexed,

SO, does anyone know how to get round this, your help will be much appreciated.

Charter · 03-10-2004, 10:03 AM

Hi. Perhaps set an htaccess file with the following line at the top of the file:

Code:

Options -Indexes

bigals · 03-10-2004, 03:21 PM

hi, where would i put this file and what do i call it, im not familiar with htaccess,

also is that the only content that would be needed in the htaccess file?

Charter · 03-10-2004, 04:49 PM

Hi. You could try adding something like ^/dir/to/ban/$ in the BANNED constant in the config file. Otherwise, try making a file named .htaccess containing Options -Indexes on one line and stick in your web root directory, assuming your OS/setup allows for htaccess files.

bigals · 03-11-2004, 06:27 AM

ok, cheers, i added the .htaccess file and it seemed to work, but now my spider only goes one link deep into the site which means i'm only getting about 15 pages indexed, any ideas why this is happening?

bigals · 03-11-2004, 10:48 AM

right, well now its sppidering properly so thats ok, but its still displaying these index pages in the serach results, what was that config file script all about and how do i add that, i don't want to cock anything up, im not very experienced with this program so any help on placing that code would be appreciated

cheers, BIGALS!

Charter · 03-11-2004, 03:14 PM

>> ...getting about 15 pages indexed, any ideas why...

Hi. When directory listing is on, there are links to all of the files. With directory listing set to off, only those links found in your site would be crawled.

>> ...its still displaying these index pages in the serach...

Perhaps an easier way would be to go to the admin panel, choose the site, click the update button, click a blue arrow on the left until you see the '-' entry on the right, and then click the red X on the right next to the '-' to delete them.

bigals · 03-11-2004, 04:30 PM

i have about 1500 of those little '-' indexes so i'm not going to delet them one by one.

That config script you gave me means i'd have to set each index to be banned too, is that correct?, any ideas now?

i can't beleive no one else has this problem, who in their right mind wants index pages to be indexed???

Charter · 03-11-2004, 05:33 PM

Hi. I didn't realize your site was so big.

Yes, that would be a very tedious process. Anyway, not tested much but perhaps try the following.

Make a file called cleanup_dashes.php with the below content, stick it in the PhpDig admin directory, and call it from the browser. Once done, run the other cleans to shore up the engine (remove orphan keywords, etcetera).

PHP Code:


			
<?php



echo "<html><body>";

$count = 0;

$relative_script_path = '..';



include "$relative_script_path/includes/config.php";

include "$relative_script_path/libs/auth.php";

include "$relative_script_path/admin/robot_functions.php";



$query = mysql_query("SELECT spider_id FROM ".PHPDIG_DB_PREFIX."spider WHERE file = '';");



while ($row = mysql_fetch_array($query)) {

  mysql_query("DELETE FROM ".PHPDIG_DB_PREFIX."engine WHERE spider_id=".$row['spider_id'].";");

  mysql_query("DELETE FROM ".PHPDIG_DB_PREFIX."spider WHERE spider_id=".$row['spider_id'].";");

  phpdigDelText($relative_script_path,$spider_id);

  $count++;

  echo $count . "<br>\n";

}



echo "<br>Done. <a href=\"index.php\" target=\"_top\">[Back]</a> to admin interface.";

echo "</body></html>";



?>

Remember to remove any "word" wrapping in the above code.

bigals · 03-12-2004, 12:37 AM

Right it tried that but it keeps returning the error:

cannot modify header info, headers already sent by auth.php
...and this french afterwards -
Vous ne pouvez accÃ©der Ã* cette page

(The header errors are longer than that its just i'm abreviating them)

also the '-' symbol is not contained within the script you sent me so how is the script finding and removing the '-' entries?

$query = mysql_query("SELECT spider_id FROM ".PHPDIG_DB_PREFIX.

in the querie it states:
"spider WHERE file = ' ';");

shouldn't it be:
"spider WHERE file = '-'");

i don't know its just i'm trying to figure out the script.

Charter · 03-12-2004, 12:57 AM

Hi. The username and password should be the ones in the config.php file. Otherwise just comment that include auth.php line out, but make sure to protect the admin directory so nobody else can run those scripts. Also, the query should be just file = '' (two single quotes, no space inbetween). The '-' is what you see onscreen, not what is in the table.

bigals · 03-12-2004, 01:16 AM

My Friend, you are a star!!!

That worked first time, brilliant, I really appreciate all the help you have given me.

My search results are fine now and I'm chuffed to bits, its looks a million times better now,

AMAZING!!!

thanks again,

BIGALS.

03-10-2004, 01:42 AM	#1
bigals Orange Mole Join Date: Nov 2003 Posts: 41	Remove Index pages Hi I have php dig set up on a web site and I'm having trouble getting round a problem. PHP DIG indexes pages when spidering, these pages are roots of folders, they appear in the spider directory as '-' entries, these are then listed when a user searches, when they are clicked they just open folder pages on my web site such as: www.domain.com/html/one/ thats a bit confusing so try this description: if you have a page called: www.domain.com/html/one/one.html the spider indexes this link but also indexes the root folder of that html page, so you get both of these results: www.domain.com/html/one/one.html www.domain.com/html/one/ I don't want these folders indexed, i just want the html pages indexed, SO, does anyone know how to get round this, your help will be much appreciated.

03-10-2004, 10:03 AM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Perhaps set an htaccess file with the following line at the top of the file: Code: Options -Indexes __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

03-10-2004, 04:49 PM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. You could try adding something like ^/dir/to/ban/$ in the BANNED constant in the config file. Otherwise, try making a file named .htaccess containing Options -Indexes on one line and stick in your web root directory, assuming your OS/setup allows for htaccess files. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

03-11-2004, 03:14 PM	#7
Charter Head Mole Join Date: May 2003 Posts: 2,539	>> ...getting about 15 pages indexed, any ideas why... Hi. When directory listing is on, there are links to all of the files. With directory listing set to off, only those links found in your site would be crawled. >> ...its still displaying these index pages in the serach... Perhaps an easier way would be to go to the admin panel, choose the site, click the update button, click a blue arrow on the left until you see the '-' entry on the right, and then click the red X on the right next to the '-' to delete them. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

03-11-2004, 04:30 PM	#8
bigals Orange Mole Join Date: Nov 2003 Posts: 41	i have about 1500 of those little '-' indexes so i'm not going to delet them one by one. That config script you gave me means i'd have to set each index to be banned too, is that correct?, any ideas now? i can't beleive no one else has this problem, who in their right mind wants index pages to be indexed??? Last edited by bigals; 03-11-2004 at 04:33 PM.

03-10-2004, 03:21 PM	#3
bigals Orange Mole Join Date: Nov 2003 Posts: 41	hi, where would i put this file and what do i call it, im not familiar with htaccess, also is that the only content that would be needed in the htaccess file?

03-11-2004, 06:27 AM	#5
bigals Orange Mole Join Date: Nov 2003 Posts: 41	ok, cheers, i added the .htaccess file and it seemed to work, but now my spider only goes one link deep into the site which means i'm only getting about 15 pages indexed, any ideas why this is happening?

03-11-2004, 10:48 AM	#6
bigals Orange Mole Join Date: Nov 2003 Posts: 41	right, well now its sppidering properly so thats ok, but its still displaying these index pages in the serach results, what was that config file script all about and how do i add that, i don't want to cock anything up, im not very experienced with this program so any help on placing that code would be appreciated cheers, BIGALS!

03-11-2004, 05:33 PM	#9
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. I didn't realize your site was so big. Yes, that would be a very tedious process. Anyway, not tested much but perhaps try the following. Make a file called cleanup_dashes.php with the below content, stick it in the PhpDig admin directory, and call it from the browser. Once done, run the other cleans to shore up the engine (remove orphan keywords, etcetera). PHP Code: <?php echo "<html><body>"; $count = 0; $relative_script_path = '..'; include "$relative_script_path/includes/config.php"; include "$relative_script_path/libs/auth.php"; include "$relative_script_path/admin/robot_functions.php"; $query = mysql_query("SELECT spider_id FROM ".PHPDIG_DB_PREFIX."spider WHERE file = '';"); while ($row = mysql_fetch_array($query)) { mysql_query("DELETE FROM ".PHPDIG_DB_PREFIX."engine WHERE spider_id=".$row['spider_id'].";"); mysql_query("DELETE FROM ".PHPDIG_DB_PREFIX."spider WHERE spider_id=".$row['spider_id'].";"); phpdigDelText($relative_script_path,$spider_id); $count++; echo $count . "<br>\n"; } echo "<br>Done. <a href=\"index.php\" target=\"_top\">[Back]</a> to admin interface."; echo "</body></html>"; ?> Remember to remove any "word" wrapping in the above code. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

03-12-2004, 12:37 AM	#10
bigals Orange Mole Join Date: Nov 2003 Posts: 41	Right it tried that but it keeps returning the error: cannot modify header info, headers already sent by auth.php ...and this french afterwards - Vous ne pouvez accÃ©der Ã* cette page (The header errors are longer than that its just i'm abreviating them) also the '-' symbol is not contained within the script you sent me so how is the script finding and removing the '-' entries? $query = mysql_query("SELECT spider_id FROM ".PHPDIG_DB_PREFIX. in the querie it states: "spider WHERE file = ' ';"); shouldn't it be: "spider WHERE file = '-'"); i don't know its just i'm trying to figure out the script.

03-12-2004, 12:57 AM	#11
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. The username and password should be the ones in the config.php file. Otherwise just comment that include auth.php line out, but make sure to protect the admin directory so nobody else can run those scripts. Also, the query should be just file = '' (two single quotes, no space inbetween). The '-' is what you see onscreen, not what is in the table. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

03-12-2004, 01:16 AM	#12
bigals Orange Mole Join Date: Nov 2003 Posts: 41	My Friend, you are a star!!! That worked first time, brilliant, I really appreciate all the help you have given me. My search results are fine now and I'm chuffed to bits, its looks a million times better now, AMAZING!!! thanks again, BIGALS.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
How to index other pages but not farther from them?	WebSpider	Mod Requests	5	02-07-2005 07:18 PM
how to index only not indexed pages?	zaartix	How-to Forum	2	07-14-2004 05:23 AM
How do you index dynamice pages?	orbitalz	How-to Forum	2	05-10-2004 05:06 PM
converted from html pages to php pages now no pages will index!!! help!!	bigals	Troubleshooting	24	04-01-2004 10:34 AM
do not index all pages	robilix	Troubleshooting	2	11-25-2003 02:50 PM