It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty. Also, if you happen to have sensitive data on your site that you do not want the world to see, you will also prefer that search engines do not index these pages (although in this case the only sure way for not indexing sensitive data is to keep it offline on a separate machine). Additionally, if you want to save some bandwidth by excluding images, stylesheets and javascript from indexing, you also need a way to tell spiders to keep away from these items

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize web sites. Not all robots cooperate with the standard; email harvesters, spambots, malware, and robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

Robot.txt is a text file.It is used when the web page is new and there is no content in the web page, therefore it is important for SEO.

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

Robots.txt is a file associated with your website used to ask different web crawlers to crawl or not crawl portions of your website.

It is a type of text file which tells bot what to crawl or what to not ?

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site.

Robots.txt is a text file which will instruct search engine machine to crawl and index pages on your website.

This file is used by search engine for crawling website's page and for given them index.

A robot.txt file instructs search bots about the pages that has to be indexed.

Seo in dubai