Robots rules
 

The robots.txt is a kind of deposited in the root directory of the web site under text files, used to tell the search engine's reptiles, which of the site content is should not be the search engine's index, which can be the index. It is commonly believed that the robots.txt file used to search engine on the target page grab.