Site Owners Forums - Webmaster Forums - View Single Post

amitmax · 03-06-2012, 04:06 AM

Very true.Generally we use it to save unusual crawling of spider on those pages having duplicate content or files which we don't want to share publicly.
Thanks

03-06-2012, 04:06 AM	#4
amitmax Registered User Join Date: Aug 2010 Posts: 215	Very true.Generally we use it to save unusual crawling of spider on those pages having duplicate content or files which we don't want to share publicly. Thanks