What is the use of Robots.txt? - Site Owners Forums

omkarsoft · 09-11-2014, 10:42 PM

Hello Friends,

Please tell me about, What is the use of Robots.txt?

MarleeNieves · 08-26-2015, 10:03 PM

Robots.txt is a text (not html) file you add on your website to let search robots know which pages you would like them not to visit.

isupersport · 08-27-2015, 01:51 AM

It is a text file where you can allow or disallow folder file and image.

smmexpertss · 10-15-2015, 02:48 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

NancyFernandes · 10-23-2015, 03:17 AM

Quote:

Originally Posted by omkarsoft

Hello Friends,

Please tell me about, What is the use of Robots.txt?

Robots.txt helps to tell bots, which pages of the website need to be crawled and which should be avoided. Robots tag is the alternate of Robots.txt file, the only difference is tag need to be mentioned in each and every web page of a website, whereas .txt file is uploaded only once i.e at the root of the website.

fhppy · 12-09-2015, 06:42 AM

robot-vacuums.

Simon s · 12-13-2015, 11:49 PM

I just want to know which is most important search engine in India like Google being and yahoo can any body tell me ?

alexparker · 12-14-2015, 03:05 AM

Robot.txt or Robots exclusion standard or robots exclusion protocol is standard used by sites to communicate web crawlers and other web robots. Site owners use robots.txt file to give instructions about their site to web robots.

senior · 12-15-2015, 02:36 AM

The robots exclusion protocol (REP), or robots.

alicemark15 · 12-16-2015, 10:44 PM

Robots.txt is a text file where user or site owner uses to make allow or disallow respective folder file and image.

aajokhio · 12-17-2015, 12:28 PM

Robots.txt file which running 24/7 and crawl websites pages. First of All, you make a file namely robots.txt after that you can type with codes and you can told search engine robots allows or disallow to crawl your site indexes.

Furthermore, If you don't understand it please visit Robots.txt Video Tutorial you can understood as well.

senior · 01-05-2016, 06:03 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

sudeepkhana · 02-29-2016, 12:03 AM

Robot.txt file uses to tell the search engine spider, what are the file to be a boot or crawl and index.

09-11-2014, 10:42 PM	#1
omkarsoft Registered User Join Date: Sep 2014 Location: Bangalore,Karnataka Posts: 133	What is the use of Robots.txt? Hello Friends, Please tell me about, What is the use of Robots.txt? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

08-26-2015, 10:03 PM	#2
MarleeNieves Registered User Join Date: May 2014 Posts: 125	Robots.txt is a text (not html) file you add on your website to let search robots know which pages you would like them not to visit. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

08-27-2015, 01:51 AM	#3
isupersport Registered User Join Date: Jun 2015 Location: New Delhi India Posts: 25	It is a text file where you can allow or disallow folder file and image. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-09-2015, 06:42 AM	#6
fhppy Registered User Join Date: Nov 2015 Posts: 61	robot-vacuums. Last edited by fhppy; 12-09-2015 at 06:42 AM.. Reason: robot

12-13-2015, 11:49 PM	#7
Simon s Registered User Join Date: Nov 2015 Posts: 17	I just want to know which is most important search engine in India like Google being and yahoo can any body tell me ? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-15-2015, 02:48 AM	#4
smmexpertss Registered User Join Date: Sep 2015 Location: UK Posts: 447	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

12-14-2015, 03:05 AM	#8
alexparker Registered User Join Date: Aug 2015 Location: 271/Kha Malibagh, Dhaka 1217, Bangladesh Posts: 14	Robot.txt Robot.txt or Robots exclusion standard or robots exclusion protocol is standard used by sites to communicate web crawlers and other web robots. Site owners use robots.txt file to give instructions about their site to web robots. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-15-2015, 02:36 AM	#9
senior Registered User Join Date: Jan 2014 Location: 150 North Radnor Chester Road, Suite F-200, Radnor, PA 19087 Posts: 178	The robots exclusion protocol (REP), or robots. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-16-2015, 10:44 PM	#10
alicemark15 Registered User Join Date: Oct 2015 Posts: 95	Robots.txt is a text file where user or site owner uses to make allow or disallow respective folder file and image. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-17-2015, 12:28 PM	#11
aajokhio Registered User Join Date: May 2015 Location: Hyderabad/Pakistan Posts: 25	Robots.txt file which running 24/7 and crawl websites pages. First of All, you make a file namely robots.txt after that you can type with codes and you can told search engine robots allows or disallow to crawl your site indexes. Furthermore, If you don't understand it please visit Robots.txt Video Tutorial you can understood as well. Last edited by aajokhio; 12-17-2015 at 12:31 PM.. Reason: Send a Video Guide Line with Robots Txt File.

01-05-2016, 06:03 AM	#12
senior Registered User Join Date: Jan 2014 Location: 150 North Radnor Chester Road, Suite F-200, Radnor, PA 19087 Posts: 178	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: " means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-29-2016, 12:03 AM	#13
sudeepkhana Registered User Join Date: Jan 2016 Location: India Posts: 1,258	Robot.txt file uses to tell the search engine spider, what are the file to be a boot or crawl and index. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM