What is robot.txt file and what are the benefits of using it?
 

Hello,

What is robot.txt file and what are the benefits of using it?

Thank you

Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not.

Best schools in Chandigarh Best schools in Mohali Best schools in Panchkula Best schools in Tricity Colleges in Chandigarh Colleges in Mohali Colleges in Tricity Colleges in Panchkula

The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.

Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not.

you can search on google..many thing for you

The explanation about robot.txt files are satisfactory to me and thanks to the concerned.

Robot.txt is a text file kept on the web servers. The robot.txt allows the site admin to permit/allow which Search engines(like google, yahoo,bing) can crawl the site. Some times the admins won't allow any of the search engines to crawl at all.

The robots.txt file is a simple text file (no html) that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip. Many webmasters utilize this file to help the search engines index the content of their websites.
If webmasters can tell the search engine spiders to skip pages that they do not consider important enough to be crawled (eg. printable versions of pages, .pdf files etc.), then they have a better opportunity to have their most valuable pages featured in the search engine results pages.The robots.txt file is a simple method of essentially easing the process for the spiders to return the most relevant search results.
Another benefit of having a robots.txt is that you can specify the location of the Google .xml or Yahoo sitemap. This also increases spiderability for the search engines.

Robots.txt file is important for seo, search engine spiders first see that where your robots.txt file because they do not know which file crawl or which not. In robots.txt file google know which file crawl.

nice and awesome
IPL 9 - Indian Premier League 2016
IPL Auction 2016 Live Results Shane Watson And Yuvraj Sold On Highest Bidding

The robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

Robots.txt file is a file through which we give instructions to the search engines whether to crawl or not the particular web page.

Its a file that is used by search engines to find out that which pages in your website you want to crawled.

Note : do not edit the robot.txt yourself if you do not know how to modify it otherwise you good pages might get blocked from search engine.

You can also use site:www.yousite.com in google to find what are the pages that are currently been crawled from your website.

Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.

The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.