indexing defined domain suffix
 

Quite a simple question really.

Is there anyway to limit the spidering to domain suffix that end .co.uk, .me.uk, gov.uk etc?

Hi Charter,

I have tried that, but it still appears to be allowing .com/.net web sites etc to be spidered and indexed :/

This is what I have got (I am using 1.8.7).

define('FORBIDDEN_EXTENSIONS','\.(net|com|rm|ico|cab|swf|css|gz|z|tar|zip|t gz|msi|arj|zoo|rar|r[0-9]+|exe|bin|pkg|rpm|deb|bz2)$');

Oops, forget what I said, as FORBIDDEN_EXTENSIONS works on path/file, not domain name. Did you mod PhpDig to allow it to index across different domains?