what is robot.txt in blogger..?
 

Hello friends,

what is robot.txt in blogger..?

Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
Basic format of the robot.txt file.
User-agent: [user-agent name]
Disallow: [URL string not to be crawled]

Creating a robots.txt file to position a blog is a simple task and if you thought that this file will not help you at all you are wrong. Since this file is very important for search engines such as Google search or yahoo search, since to be able to index your website and that it can be shown in the search results the Robots of these companies look in your file Robots.txt for the different established parameters where we will place which parts of our website are available to the public or not, plus we will give you the exact location of our site map.

Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.

Allowing all web crawlers access to all content

User-agent: *
Disallow:

Using this syntax in a robots.txt file tells web crawlers to crawl all pages on www.example.com, including the homepage.

Robots.txt is a text file on the server in that you can write which directories and web pages or links should not be included for search results. It means you can restrict search engine bots to crawl some directories and web pages or links of your website or blog. Now custom robots.txt is available for Blogger.

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

Robots.txt it’s a text file, it's instructed to the search engine which pages on your site to crawl.