Site Owners Forums - Webmaster Forums - View Single Post

lisaryan · 09-19-2016, 05:11 AM

The robots.txt file helps search engines understand which pages on a website should be indexed and crawled. This file can also be used to block the unwanted crawlers to search your websites.

09-19-2016, 05:11 AM	#11
lisaryan Registered User Join Date: Sep 2016 Location: Bangalore Posts: 58	The robots.txt file helps search engines understand which pages on a website should be indexed and crawled. This file can also be used to block the unwanted crawlers to search your websites.