View Single Post
Old 04-03-2004, 12:24 AM   #1
peterpeter
Green Mole
 
Join Date: Mar 2004
Posts: 7
Speed up spidering by skipping internal page links

As I explained in this thread, spidering can be very slow due to the existence of (many) internal page links, such as the <A HREF="#1070721880">xxx</A> and <A NAME="1070721880"></A> pair. Since such links don't serve any purpose for the spider functionality, I suggest to skip spidering these links.

Peter
peterpeter is offline   Reply With Quote