spidering links but not their text

haxored · 01-13-2004, 06:41 AM

I've got a bunch of pages with a navigation menu on it, as well as some text I don't want indexed.

I want to exclude everything but a <div> tag that contains all the text I want to index.. However...

it seems that if I do that, all of the links in my navigation menu aren't spidered. That is not helpful.

How can I get phpDig to spider my entire site, but only index one section of every page?

Charter · 01-14-2004, 10:32 AM

Hi. You might try using PHPDIG_EXCLUDE_COMMENT and PHPDIG_INCLUDE_COMMENT from the config file, each on their own line, to exclude a portion of a page. Depending on your navigation menu, it might not be getting indexed because PhpDig excludes certain tags from index. If this is the case, you might try setting up a simple HTML page with links to your site, and then crawl this page. Once the crawl is done, you can delete the simple page from the admin panel.

MaXius · 01-27-2004, 02:23 AM

I have a similar problem,

I have a pulldown menu, (div tags) and phpdig pulls all the menu options into the search results, hence most words you would search for are on alll pages. in a big ugly mess.

Does phpdig still crawl the links within a PHPDIG_EXCLUDE_COMMENT area even tho it doesnt put these into its database?

Thanks

Charter · 01-27-2004, 09:58 AM

Hi. The exclude/include comments are for omitting parts of a page from indexing.

MaXius · 01-27-2004, 12:49 PM

Yeh, gathered that... does it still spider the links within an excluded part tho...?

Ta

Charter · 01-27-2004, 05:01 PM

Hi. Using the below simple example, PhpDig works as follows:

Code:

<html>
<body>
This text is indexed
<!-- phpdigExclude -->
<a href="http://www.this_link.com/is_followed.html">This text is ignored</a>
This text is also ignored
<!-- phpdigInclude -->
</body>
</html>

To change this behavior, the phpdigExplore and/or phpdigIndexFile functions in robot_functions.php could be modified.

01-13-2004, 06:41 AM	#1
haxored Green Mole Join Date: Jan 2004 Posts: 1	spidering links but not their text I've got a bunch of pages with a navigation menu on it, as well as some text I don't want indexed. I want to exclude everything but a <div> tag that contains all the text I want to index.. However... it seems that if I do that, all of the links in my navigation menu aren't spidered. That is not helpful. How can I get phpDig to spider my entire site, but only index one section of every page?

01-14-2004, 10:32 AM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. You might try using PHPDIG_EXCLUDE_COMMENT and PHPDIG_INCLUDE_COMMENT from the config file, each on their own line, to exclude a portion of a page. Depending on your navigation menu, it might not be getting indexed because PhpDig excludes certain tags from index. If this is the case, you might try setting up a simple HTML page with links to your site, and then crawl this page. Once the crawl is done, you can delete the simple page from the admin panel. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

01-27-2004, 09:58 AM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. The exclude/include comments are for omitting parts of a page from indexing. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

01-27-2004, 05:01 PM	#6
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Using the below simple example, PhpDig works as follows: Code: <html> <body> This text is indexed <!-- phpdigExclude --> <a href="http://www.this_link.com/is_followed.html">This text is ignored</a> This text is also ignored <!-- phpdigInclude --> </body> </html> To change this behavior, the phpdigExplore and/or phpdigIndexFile functions in robot_functions.php could be modified. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Change phpdigExclude to not spider Links as well as text	gkdeveloper	How-to Forum	0	09-12-2007 04:54 AM
Exclude links from indexing, keep text	digger_123	How-to Forum	0	12-20-2006 05:14 AM
Text ads and sponsored links?	WebSpider	Mod Requests	2	03-09-2005 01:17 AM
Text Links Showing up in Keywords List	DannyD	How-to Forum	2	09-24-2004 11:02 PM
Spidering....links found : 0	-IAN-	Troubleshooting	23	08-25-2004 05:53 PM

01-27-2004, 02:23 AM	#3
MaXius Green Mole Join Date: Jan 2004 Location: Melbourne, Australia Posts: 2	I have a similar problem, I have a pulldown menu, (div tags) and phpdig pulls all the menu options into the search results, hence most words you would search for are on alll pages. in a big ugly mess. Does phpdig still crawl the links within a PHPDIG_EXCLUDE_COMMENT area even tho it doesnt put these into its database? Thanks

01-27-2004, 12:49 PM	#5
MaXius Green Mole Join Date: Jan 2004 Location: Melbourne, Australia Posts: 2	Yeh, gathered that... does it still spider the links within an excluded part tho...? Ta