What is robot.txt file and what are the benefits of using it? - Page 3 - Site Owners Forums

luffy268 · 12-19-2016, 03:11 AM

Robots.txt is a text file which has few lines and this file interacts with all kind of crawlers or spiders like Googlebot, which is Google search engine's spider.

Search Engine always want to index good and fresh contain on the web and these robot.txt instruct the web crawlers to how to index and crawl your blog in the search results.

Remember,Crawler or spider will firstly look at your robots.txt files to obey the rules you have instructed It's mean you can restrict any web page on your blog from web crawlers so that it can’t get indexed in search engines like your blog your demo page, labels page, Achieve or any other pages that are not as important to get indexed.

Nandu41 · 12-21-2016, 02:46 AM

Robots.txt is a text file which has few lines and this file interacts with all kind of crawlers or spiders like Googlebot, which is Google search engine's spider.

Search Engine always want to index good and fresh contain on the web and these robot.txt instruct the web crawlers to how to index and crawl your blog in the search results.

12-19-2016, 03:11 AM	#31
luffy268 Registered User Join Date: Dec 2016 Posts: 94	Robots.txt is a text file which has few lines and this file interacts with all kind of crawlers or spiders like Googlebot, which is Google search engine's spider. Search Engine always want to index good and fresh contain on the web and these robot.txt instruct the web crawlers to how to index and crawl your blog in the search results. Remember,Crawler or spider will firstly look at your robots.txt files to obey the rules you have instructed It's mean you can restrict any web page on your blog from web crawlers so that it can’t get indexed in search engines like your blog your demo page, labels page, Achieve or any other pages that are not as important to get indexed. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

12-21-2016, 02:46 AM	#32
Nandu41 Registered User Join Date: Sep 2016 Posts: 65	Robots.txt is a text file which has few lines and this file interacts with all kind of crawlers or spiders like Googlebot, which is Google search engine's spider. Search Engine always want to index good and fresh contain on the web and these robot.txt instruct the web crawlers to how to index and crawl your blog in the search results. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)