Its a file that is used by search engines to find out that which pages in your website you want to crawled.
Note : do not edit the robot.txt yourself if you do not know how to modify it otherwise you good pages might get blocked from search engine.
You can also use
site:www.yousite.com in google to find what are the pages that are currently been crawled from your website.