Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Robot.txt :- Robot.txt is also known as the robots exclusion protocol(REP),is a text file webmaster create to instruct robots( typically search engine robots) how to crawl and index pages on their website. It is used to the new website when there is no content.

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.

Robots.txt is a text file. it gives instruction to bots to crawlers about indexing and caching of a website or webpage.

The robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results.*Robots.txt*files are useful: If you want search engines to ignore any duplicate pages on your website.

robots.txt is a text file, it indicates the crawler to which to crawl and which one don't want to crawl.

Bots will use robots.txt to crawl our website and webpages used for crawlers about indexing .

The basic use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

robots.txt is a file, that guides the crawler which one to crawl and which to not crawl..

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

The robots avoidance convention (REP), or robots.txt is a content record website admins make to educate robots (normally web search tool robots) how to creep and file pages on their site.

I first thought you wanted to ask until I realized you answered your own question.

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.