what is robots.txt file?
 

what is a robots.txt file:

Solution
 

I think best answer for your question is Wikipedia https:// en.wikipedia.org/ wiki/ Robots_exclusion_standard , please check it and let us know, if you required any further information.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

Robots.txt is a content record that rundowns pages which contain directions for web indexes robots. The record records site pages that are permitted and refused from internet searcher creeping.