Spidering issue

[cefiro]

I have yet to install phpdig for my website, but I would like to possibly prevent an issue that might arise once the spidering commenses. I have tried a trial of one piece of commercial software that spiders a site to create a search database like phpdig does. When I ran it I noticed that is was not spidering properly. Our site masks the true url. For example, http://www.clawfootsupply.com/product461 is a page on my site. The page that displays there a sort of 'product.php' that includes other php files to be the header and footer and other features. I think the fact that the url doesn't end in any sort of .php or.htm or something similar confused the spider. I noticed that the spider would be coming across http://www.clawfootsupply.com/product461/style68 or http://www.clawfootsupply.com/product461/type55 and others like that. There is not way from the product461 page to get to the sylte68 or type 55 pages, so why was the spider navigating that way? Will phpdig do this also? Those above "bad" links would actually take you to the product461 page, the end is ignored. You can obviously see the problems with spidering this way, since it would be indexing paths that a user would never come across on the site and be doing the same page multiple times. There is no way to get through to those links, as I mentioned, so it might have just been a bug in the spider and phpdig will not have a problem. Just thought I'd try to head off headaches form occuring later. If this is confusing let me know and I'll try to explain it a different way Thanks.