Spidering problem please help

KaZ · 11-30-2005, 09:50 AM

I am trying to spider certian urls like http://cgi.ebay.com/Sweetie-Pies-Bab...QQcmdZViewItem

and when it tryes to crawl it it adds a / to the end of the url which makes the url a 404 how would i fix this issue.

thanks in advance
KaZ

KaZ · 12-05-2005, 06:59 AM

This isnt advertising that is not my link it is just an example i am creating a auction search website and need to crawl auction listings from all auction sites only site im having an issue with so far is Ebay the auction listing URl's are weird and if you add a trailing slash to them it makes the link a 404 but it seems to only do it on these urls other urls like http://auctionsitetest.com/auctiondetails.php?id=222 get listed fine with no trailing slash but for some reason only the ebay links that i have encountered so far do this any suggestions please?

11-30-2005, 09:50 AM	#1
KaZ Green Mole Join Date: Jul 2004 Posts: 2	Spidering problem please help I am trying to spider certian urls like http://cgi.ebay.com/Sweetie-Pies-Bab...QQcmdZViewItem and when it tryes to crawl it it adds a / to the end of the url which makes the url a 404 how would i fix this issue. thanks in advance KaZ

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Spidering problem	mark40	Troubleshooting	1	08-28-2007 04:06 AM
Problem with spidering	tomjed	Troubleshooting	0	02-09-2006 02:50 AM
Problem Spidering	Trallis	Troubleshooting	6	11-02-2005 07:58 AM
Problem Spidering	jmitchell	Troubleshooting	3	12-29-2004 05:42 PM
spidering problem	nathansc	How-to Forum	3	06-17-2004 03:25 PM

12-05-2005, 06:59 AM	#2
KaZ Green Mole Join Date: Jul 2004 Posts: 2	This isnt advertising that is not my link it is just an example i am creating a auction search website and need to crawl auction listings from all auction sites only site im having an issue with so far is Ebay the auction listing URl's are weird and if you add a trailing slash to them it makes the link a 404 but it seems to only do it on these urls other urls like http://auctionsitetest.com/auctiondetails.php?id=222 get listed fine with no trailing slash but for some reason only the ebay links that i have encountered so far do this any suggestions please?