Go for rustyblogger.com get full details of robot.txt there.

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

Robots.txt is text file inserted into the website to contain instructions for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling. They help to control the search engine robots in your website.

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Thanks for sharing this information.

Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

robots.txt it will be use for search engines bots instruction to index and noindex the site and subfolder of the site. webmaster can stop pages and files through robots.txt