Indexing single page as its own site

bloodjelly · 10-13-2005, 11:16 AM

Lots of dynamically-generated websites offer a good deal of content by changing variables in the URI string. It'd be neat if there was a way to index these pages as if they're a site.

An example is Myspace. Users aren't viewed by subdirectory or subdomain, but by a "friendID" variable, like this: http://profile.myspace.com/index.cfm...endID=28915943

The California Gas Prices website also has a similar structure, with URIs like this: http://www.californiagasprices.com/i...AN%20FRANCISCO

Now, these pages change enough on their own that it'd be nice to index them as a site rather than just pages in a site (so that they could be searched separately.)

Is this possible?

Charter · 10-14-2005, 10:06 AM

PhpDig is currently domain based for indexing with a storage scheme as in this post. There are spots in the code that specifically bust up a link, and then parts of the link are stored in the tables. It is possible that an option could be added to not do this, whether for queries or directories, and treat the link as its own site, but then other links on the page become an issue. Right now the domain is the focal point, and links relate to the domain. If the focal point is changed to a specific link in a domain, rather than the domain itself, other links in the site won't relate to it and would probably need to be ignored.

bloodjelly · 10-14-2005, 10:44 AM

Hi Charter,

Thanks for your response. What you're saying completely makes sense, but couldn't you have phpDig treat the individual page as a site, and then only follow links that shared a vital characteristic? (If friendID=x, if area=san_francisco). Maybe the variable to track could be specified for each unique domain?

Charter · 10-15-2005, 08:39 AM

If you want to index just one page of a site, you can stick the full link, with query string and all, in the admin panel text box, set search depth to zero, links per to one, use no, and dig. If you want to restrict indexing to certain link characteristics, and you are familiar with regular expressions, check out this thread. Maybe a new field in a table for certain link characteristics would be a good addition, but right now you'd have to edit the config file each time for different sites.

10-13-2005, 11:16 AM	#1
bloodjelly Purple Mole Join Date: Dec 2003 Posts: 106	Indexing single page as its own site Lots of dynamically-generated websites offer a good deal of content by changing variables in the URI string. It'd be neat if there was a way to index these pages as if they're a site. An example is Myspace. Users aren't viewed by subdirectory or subdomain, but by a "friendID" variable, like this: http://profile.myspace.com/index.cfm...endID=28915943 The California Gas Prices website also has a similar structure, with URIs like this: http://www.californiagasprices.com/i...AN%20FRANCISCO Now, these pages change enough on their own that it'd be nice to index them as a site rather than just pages in a site (so that they could be searched separately.) Is this possible? __________________ Foundmyself.com artist community, art galleries

10-14-2005, 10:06 AM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	PhpDig is currently domain based for indexing with a storage scheme as in this post. There are spots in the code that specifically bust up a link, and then parts of the link are stored in the tables. It is possible that an option could be added to not do this, whether for queries or directories, and treat the link as its own site, but then other links on the page become an issue. Right now the domain is the focal point, and links relate to the domain. If the focal point is changed to a specific link in a domain, rather than the domain itself, other links in the site won't relate to it and would probably need to be ignored. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

10-14-2005, 10:44 AM	#3
bloodjelly Purple Mole Join Date: Dec 2003 Posts: 106	Hi Charter, Thanks for your response. What you're saying completely makes sense, but couldn't you have phpDig treat the individual page as a site, and then only follow links that shared a vital characteristic? (If friendID=x, if area=san_francisco). Maybe the variable to track could be specified for each unique domain? __________________ Foundmyself.com artist community, art galleries

10-15-2005, 08:39 AM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	If you want to index just one page of a site, you can stick the full link, with query string and all, in the admin panel text box, set search depth to zero, links per to one, use no, and dig. If you want to restrict indexing to certain link characteristics, and you are familiar with regular expressions, check out this thread. Maybe a new field in a table for certain link characteristics would be a good addition, but right now you'd have to edit the config file each time for different sites. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
successful indexing of every site but site where phpdig is served	phillystyle123	Troubleshooting	1	02-21-2005 09:06 PM
Can only index files in a single directory	gcrachel	Troubleshooting	5	09-28-2004 06:23 AM
URLs containing single quotes	mmaattttt	Troubleshooting	2	08-26-2004 09:03 PM
Reindexing site won't index certain page	gman	Troubleshooting	4	08-06-2004 01:05 PM
Remove single pdf	motopsycho	How-to Forum	2	04-30-2004 05:22 PM