Robots.txt

[farhandiesel]

A robots.txt file may be a file at the foundation of your web {site} that indicates those elements of your site you don’t need accessed by computer programme crawlers. The file uses the Robots Exclusion commonplace, that may be a protocol with a tiny low set of commands which will be accustomed indicate access to your web site by section and by specific styles of internet crawlers (such as mobile crawlers vs desktop crawlers).