|
The spiders exemption conventional, also known as the spiders exemption method or simply spiders.txt, is a conventional used by sites to talk with web spiders and other web spiders. The common identifies how to tell the web software about which areas of the website should not be prepared or examined.
|