Site Owners Forums - Webmaster Forums - View Single Post

George re · 09-13-2016, 04:32 AM

Robots.txt is a text file which gives the instructions to crawlers about the website that which pages should be crawled or not.

09-13-2016, 04:32 AM	#8
George re Registered User Join Date: Sep 2016 Posts: 33	Robots.txt is a text file which gives the instructions to crawlers about the website that which pages should be crawled or not.