Less hosts than I put

noel · 11-06-2005, 10:36 AM

Hi CHARTER,

I put 773 urls and after the spider stops without problems, phpdig finds just 588 host. I don't understand why, all the urls works well, all the sites exists.

Thank you

Noël

Charter · 11-06-2005, 02:08 PM

Maybe certain sites block spiders using a robots.txt or .htaccess file. Maybe the encoding could not be understood in the phpdigMakeUTF8 function. Maybe the status from the phpdigTestUrl function returned NOHOST, NOFILE, LOOP, or NEWHOST. Maybe something else.

11-06-2005, 10:36 AM	#1
noel Orange Mole Join Date: Aug 2005 Posts: 44	Less hosts than I put Hi CHARTER, I put 773 urls and after the spider stops without problems, phpdig finds just 588 host. I don't understand why, all the urls works well, all the sites exists. Thank you Noël

11-06-2005, 02:08 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Maybe certain sites block spiders using a robots.txt or .htaccess file. Maybe the encoding could not be understood in the phpdigMakeUTF8 function. Maybe the status from the phpdigTestUrl function returned NOHOST, NOFILE, LOOP, or NEWHOST. Maybe something else. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Maximum Hosts?	pgzn	The Mole Hole	0	07-13-2006 04:35 PM
Blocking particular hosts or domains?	cewyattjr	How-to Forum	0	06-09-2006 12:49 PM
FTP Put Error	cash4read	Troubleshooting	6	05-28-2004 11:45 AM
Limiting the search to certain hosts	lighthouse	Mod Requests	1	03-11-2004 03:30 PM
Put text_content in DB?	renehaentjens	Mod Requests	0	11-27-2003 07:45 AM