Site Owners Forums - Webmaster Forums - View Single Post

SamMarley · 02-25-2013, 06:10 AM

Robots.txt is a file that you can add to your website which instructs the search engine crawler which pages to crawl. You can also instruct the crawlers not to crawl some pages through robots.txt.

02-25-2013, 06:10 AM	#7
SamMarley Registered User Join Date: Feb 2013 Posts: 2	Robots.txt is a file that you can add to your website which instructs the search engine crawler which pages to crawl. You can also instruct the crawlers not to crawl some pages through robots.txt.