Site Owners Forums - Webmaster Forums - View Single Post

samaron · 01-21-2018, 10:03 PM

Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.

01-21-2018, 10:03 PM	#2
samaron Registered User Join Date: Sep 2017 Posts: 177	Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.