Site Owners Forums - Webmaster Forums - View Single Post

ronaldotson45 · 12-11-2012, 11:53 PM

Robot.txt file indicate search crawler which information in this website is prohibited or do not crawl. if owner of the site does not want any of the file to be indexed or crawled by robot or spider then robot.txt file being used

12-11-2012, 11:53 PM	#15
ronaldotson45 Registered User Join Date: Dec 2012 Posts: 5	Robot.txt file indicate search crawler which information in this website is prohibited or do not crawl. if owner of the site does not want any of the file to be indexed or crawled by robot or spider then robot.txt file being used