Crawl for illegal content

losttoy · 12-24-2004, 08:03 AM

Hi,

I need to crawl the sites of customers hosted with us and try to assess if any of them have illegal/pornographic material hosted on their servers.

I am currently trying to use Metis but it is horribly error prone and slooowww.

I want a site to be crawled for certain keywords, if content of any URL matches the keywords then I want a report of all such URLs for a customer to be emailed to my support-desk for verification/action.

Has anyone done this with PHPDig? Is this possible or is PHPDig going to give me reports only when I query it? More like I run a query on the indexed information everyday and mail myself a report.

Thanks,

Siddhartha

12-24-2004, 08:03 AM	#1
losttoy Green Mole Join Date: Dec 2004 Posts: 1	Crawl for illegal content Hi, I need to crawl the sites of customers hosted with us and try to assess if any of them have illegal/pornographic material hosted on their servers. I am currently trying to use Metis but it is horribly error prone and slooowww. I want a site to be crawled for certain keywords, if content of any URL matches the keywords then I want a report of all such URLs for a customer to be emailed to my support-desk for verification/action. Has anyone done this with PHPDig? Is this possible or is PHPDig going to give me reports only when I query it? More like I run a query on the indexed information everyday and mail myself a report. Thanks, Siddhartha

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
never ending crawl	oehk	Troubleshooting	0	12-12-2006 06:34 AM
CRAWL - SPDIER A WHOLE WEBSITE	mahesr	The Mole Hole	0	11-01-2006 10:43 PM
No results after the crawl	GunMuse	Script Installation	8	04-12-2005 06:57 PM
crawl my website only	hula	Troubleshooting	0	08-29-2004 04:37 AM
Crawl mutiple urls	Jeff	How-to Forum	4	02-03-2004 07:59 PM