What is robots.txt used for? - Site Owners Forums

danielnash · 03-09-2016, 12:31 AM

I learnt here about sitemap.xml file. Now I'm little bit confused for what is robots.txt used for?

Please guys clear it.

RH-Calvin · 03-10-2016, 01:07 AM

Robots.txt is a text file that is used to insert into the website. This file contains URLs that are allowed and disallowed from search engine crawling.

sudeepbhati · 03-12-2016, 04:48 AM

The purpose of the using robots.txt file is to tell the search engine bots which file should and which should not be indexed by them. In order to allow search bots to crawl and index the entire content of your website.

sudeepkhana · 03-14-2016, 12:01 AM

Robot.txt file Is used to tell the search engine which Page you do not Want Crawled or Indexed.

stuartspindlow9 · 03-14-2016, 05:13 AM

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.

jgaldnik · 03-14-2016, 12:53 PM

Google search bro

bradvictor1 · 03-14-2016, 11:25 PM

Its very easy to understand here we give instructions to search engine which page they can crawl and index. And get more detail from online blogs.

03-09-2016, 12:31 AM	#1
danielnash Registered User Join Date: Sep 2014 Location: california Posts: 52	What is robots.txt used for? I learnt here about sitemap.xml file. Now I'm little bit confused for what is robots.txt used for? Please guys clear it. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-10-2016, 01:07 AM	#2
RH-Calvin Registered User Join Date: May 2015 Location: Forum Posts: 2,517	Robots.txt is a text file that is used to insert into the website. This file contains URLs that are allowed and disallowed from search engine crawling. __________________ █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| $1 VPS Hosting █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Windows Server with Remote Desktop Access █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Servers with Free IPMI Setup

03-12-2016, 04:48 AM	#3
sudeepbhati Registered User Join Date: Feb 2016 Posts: 20	The purpose of the using robots.txt file is to tell the search engine bots which file should and which should not be indexed by them. In order to allow search bots to crawl and index the entire content of your website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-14-2016, 12:01 AM	#4
sudeepkhana Registered User Join Date: Jan 2016 Location: India Posts: 1,258	Robot.txt file Is used to tell the search engine which Page you do not Want Crawled or Indexed. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-14-2016, 11:25 PM	#7
bradvictor1 Registered User Join Date: Jun 2014 Location: India Posts: 127	Its very easy to understand here we give instructions to search engine which page they can crawl and index. And get more detail from online blogs. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-14-2016, 05:13 AM	#5
stuartspindlow9 Registered User Join Date: Jan 2016 Location: Thailand Posts: 32	Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.

03-14-2016, 12:53 PM	#6
jgaldnik Registered User Join Date: Mar 2016 Posts: 256	Google search bro

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM