Site Owners Forums - Webmaster Forums - View Single Post

Hardeep_Singh · 02-17-2016, 09:52 PM

Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

or

Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

02-17-2016, 09:52 PM	#6
Hardeep_Singh Registered User Join Date: Feb 2016 Posts: 3	Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. or Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. Last edited by Hardeep_Singh; 02-17-2016 at 09:58 PM..