What is the use of Robots.txt ? - Site Owners Forums

jerrysmith · 10-26-2016, 05:40 AM

Hello friends,

I want to know that What is the use of Robots.txt ? tell me ..

valentinebruce9 · 10-26-2016, 11:02 PM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

jamesvincent21 · 10-27-2016, 02:14 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots.The "Disallow: /" tells the robot that it should not visit any pages on the site.

jasonroy21 · 10-27-2016, 02:15 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots.The "Disallow: /" tells the robot that it should not visit any pages on the site.

abhishek1 · 10-27-2016, 03:34 AM

Robot.txt files are used to give information to the spiders to visit the website for crawling and indexing.We can use Disallow option which will tell the robot not to visit the website.

whmcssmarters · 10-27-2016, 05:45 AM

Quote:

Originally Posted by valentinebruce9

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

Hi, yes right I am agree with you.

akshara123 · 10-27-2016, 06:06 AM

Robots.txt is nothing hide the some pages of our site because of privacy.for example you have an e-commerce site like shopclues, flipkart in festival time we post some offer in particular page after u no need but it can use future purpose in that situation also we can use.

rk484 · 10-27-2016, 06:39 AM

It's a file used by the webmasters to give instructions to web robots. It helps if we don't want any page to be crawled by any web robot.

timly · 10-27-2016, 06:50 AM

Robots.txt is a text file,we can write in word and notepad. Robots.txt you put on your website to tell the search engine which pages you would like them not to visit.

samadny · 10-27-2016, 07:08 PM

It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty. Also, if you happen to have sensitive data on your site that you do not want the world to see, you will also prefer that search engines do not index these pages (although in this case the only sure way for not indexing sensitive data is to keep it offline on a separate machine). Additionally, if you want to save some bandwidth by excluding images, stylesheets and javascript from indexing, you also need a way to tell spiders to keep away from these items.

ethanmichael201 · 11-01-2016, 11:04 PM

Web site owners use the /robots.txt file to give instructions about their site to web robots.The "Disallow: /" tells the robot that it should not visit any pages on the site that is called for use robot.txt

stuartsmithh5 · 11-03-2016, 02:35 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

mjohn271 · 11-03-2016, 05:17 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

tenny12 · 11-03-2016, 05:39 AM

robots.txt is a ON PAGE SEO TECHNIQUE is used to hide your data during the google reading on your website.
it has two choice Allow and Disallow if give ur data Allow site google read your data otherwise it can't read your data

fishpawnbroker · 11-03-2016, 06:16 AM

When a search engine crawls (visits) your website, the first thing it looks for is your robots.txt file. This file tells search engines what they should and should not index.

10-26-2016, 05:40 AM	#1
jerrysmith Registered User Join Date: Oct 2015 Posts: 36	What is the use of Robots.txt ? Hello friends, I want to know that What is the use of Robots.txt ? tell me .. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-26-2016, 11:02 PM	#2
valentinebruce9 Registered User Join Date: Aug 2016 Posts: 195	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: " means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

10-27-2016, 02:14 AM	#3
jamesvincent21 Registered User Join Date: Sep 2016 Posts: 326	Web site owners use the /robots.txt file to give instructions about their site to web robots.The "Disallow: /" tells the robot that it should not visit any pages on the site. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

10-27-2016, 02:15 AM	#4
jasonroy21 Registered User Join Date: Jul 2016 Posts: 755	Web site owners use the /robots.txt file to give instructions about their site to web robots.The "Disallow: /" tells the robot that it should not visit any pages on the site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-27-2016, 03:34 AM	#5
abhishek1 Registered User Join Date: Oct 2016 Posts: 71	Robot.txt files are used to give information to the spiders to visit the website for crawling and indexing.We can use Disallow option which will tell the robot not to visit the website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-27-2016, 06:06 AM	#7
akshara123 Registered User Join Date: Sep 2016 Location: chennai Posts: 93	Robots.txt is nothing hide the some pages of our site because of privacy.for example you have an e-commerce site like shopclues, flipkart in festival time we post some offer in particular page after u no need but it can use future purpose in that situation also we can use. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. logo designers chennai To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. brochure designers chennai To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. Love marriage procedure in Chennai, How to register marriage in Chennai, Court marriage procedure in Chennai To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. HRD Attestation in Chennai, Embassy Attestation, Apostalle Services To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. UAE Embassy attestation in bangalore

10-27-2016, 06:39 AM	#8
rk484 Registered User Join Date: Oct 2016 Posts: 37	It's a file used by the webmasters to give instructions to web robots. It helps if we don't want any page to be crawled by any web robot. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-27-2016, 06:50 AM	#9
timly Registered User Join Date: May 2016 Posts: 105	Robots.txt is a text file,we can write in word and notepad. Robots.txt you put on your website to tell the search engine which pages you would like them not to visit. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-27-2016, 07:08 PM	#10
samadny Registered User Join Date: Oct 2016 Posts: 33	It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. For instance, if you have two versions of a page (one for viewing in the browser and one for printing), you'd rather have the printing version excluded from crawling, otherwise you risk being imposed a duplicate content penalty. Also, if you happen to have sensitive data on your site that you do not want the world to see, you will also prefer that search engines do not index these pages (although in this case the only sure way for not indexing sensitive data is to keep it offline on a separate machine). Additionally, if you want to save some bandwidth by excluding images, stylesheets and javascript from indexing, you also need a way to tell spiders to keep away from these items. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-01-2016, 11:04 PM	#11
ethanmichael201 Registered User Join Date: Jun 2016 Posts: 219	Web site owners use the /robots.txt file to give instructions about their site to web robots.The "Disallow: /" tells the robot that it should not visit any pages on the site that is called for use robot.txt __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

11-03-2016, 02:35 AM	#12
stuartsmithh5 Registered User Join Date: Oct 2016 Posts: 81	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: " means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-03-2016, 05:17 AM	#13
mjohn271 Registered User Join Date: Aug 2016 Posts: 175	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: " means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-03-2016, 05:39 AM	#14
tenny12 Registered User Join Date: Nov 2016 Posts: 24	robots.txt is a ON PAGE SEO TECHNIQUE is used to hide your data during the google reading on your website. it has two choice Allow and Disallow if give ur data Allow site google read your data otherwise it can't read your data

11-03-2016, 06:16 AM	#15
fishpawnbroker Registered User Join Date: Aug 2015 Location: London Posts: 47	When a search engine crawls (visits) your website, the first thing it looks for is your robots.txt file. This file tells search engines what they should and should not index. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Robots.txt	jaysh4922	Search Engine Optimization	13	07-28-2016 05:10 AM
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM