Robot.txt?
 

What is Robot.txt? Why it is used?

robots.txt file used as directives to search engine crawlers. you can forbid crawlers to crawl specific pages of your site

Robots.txt is a text (not html) file you put on your web page to tell search robots which pages you would like them not to visit.

Yes all answers are some but not exact :-
Exactly robots.txt is a text file which has two parts
user agent and disallow
In dis allow, we place that file we don't want to crawl and index in search engine.

I think they explained you why you should keep it pn your host

robots.txt is helpful to crawls the site or not by using allow or disallow.

picnic spots near delhi | New year packages delhi 2015 | camp in kanatal

This helps to prevent a particular page to crowl that the user don't want.

Robots.txt is an text file used by search crewl.it tells crewl which part of website to index or which not index in search result.Find More At Custom Robot.txt

It is used to allow or disallow search engine to crawl your pages..

It help to instruct crawler that which page should be cache on which is not.

Robot.txt file used to block crawler to crawl some content in your website.

Robots.txt is a text file, It is used to allow or disallow search engines to crawl your website or webpage

Generate effective robots.txt files that help ensure Google and other search engines are crawling and indexing your site properly.

Robots.txt is a text file which we upload on website server to instruct spider the which page or folder should be crawl and which is not.

Thanks for your question and happy to read all friends suggestion