Site Owners Forums - Webmaster Forums - View Single Post

rajj · 12-27-2016, 11:47 PM

robots.txt is a text file and if you don't want to crawl your webpage, you just write that file name and give robot.txt extension so crawler don't crawl that page.

12-27-2016, 11:47 PM	#3
rajj Registered User Join Date: Dec 2016 Posts: 36	robots.txt is a text file and if you don't want to crawl your webpage, you just write that file name and give robot.txt extension so crawler don't crawl that page.