Site Owners Forums - Webmaster Forums - View Single Post

stephinewinkle · 01-09-2015, 01:06 AM

Robots.txt is a text file which we upload on website server to instruct spider the which page or folder should be crawl and which is not.

01-09-2015, 01:06 AM	#14
stephinewinkle Registered User Join Date: Jan 2015 Posts: 1	Robots.txt is a text file which we upload on website server to instruct spider the which page or folder should be crawl and which is not.