The robots.txt file
 

The robots.txt not command, but a agreement. Search engine in the grab a web site page, will be the first to grab the robots.txt file, so the file the meaning of existence is tell search engines what page can be grab, which page cannot be grab. When the search engine grab site web page, it will first visit the web site under the root directory of the robots files, if there is the documents, search engine will be defined in the paper the content of grab, if not, the search engines will grab all allow it can catch page.