Site Owners Forums - Webmaster Forums - View Single Post

Dennydevi · 09-19-2017, 01:54 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.

09-19-2017, 01:54 AM	#30
Dennydevi Registered User Join Date: Sep 2016 Posts: 429	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.