|
It is important to note that malicious crawlers are likely to completely ignore robots.txt and as such, this protocol does not make a good security mechanism. Only one "Disallow:" line is allowed for each URL. Each subdomain on a root domain uses separate robots.txt files.
|