what is the use of robot.txt? - Site Owners Forums

europeanoversea · 03-18-2016, 01:21 AM

what is the use of robot.txt?

alicemark15 · 03-18-2016, 01:52 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

arianagrand · 03-18-2016, 01:58 AM

robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

daviddakarai · 03-18-2016, 03:00 AM

Robots.txt is a simple text file that is placed on your site’s root directory. It is that file on your website that tells these search engine robots what to crawl and what not to crawl on your site. To facilitate auto-discovery of your sitemap file through your robots.txt, all you have to do is place a directive with the URL in your robots.txt

krify · 03-18-2016, 03:17 AM

The main use of Robot text file is to make search engines know what is are the URLs need to cowl around the websites !

robot dot text is used to upload at the time of site creation then site with sitemap upload using ftp publications.

AshokDixit89 · 03-18-2016, 06:00 AM

Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

indianlawwatch · 03-18-2016, 11:20 PM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Sunsite · 03-19-2016, 01:24 AM

It is a instruction to robots in a text how to crawl and index pages on their website

Sangeetakashyap · 03-19-2016, 04:14 AM

Robot.txt is a text file that has been used by website owners to give instructions to the web robots about the website. It helps the robots of Google to crawl each page of website so that it can index it properly & can rank it well in the Google search engine easily. It is an important factor of On Page of any website.

accutranskpo · 03-22-2016, 04:37 AM

robots.txt is a text file which is used by web-master to give crawling instructions to the search engine crawlers which page is to be crawl and which page is not.

vashikaransutra · 03-22-2016, 04:54 AM

It is must file for every website if webmasters want to do SEO. This file tells to Robots about which files should crawl or which not. It must place on root of the website.

Nexevo Technolo · 03-22-2016, 05:49 AM

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored

aajokhio · 03-22-2016, 04:40 PM

Basically, Robots Txt File Allow or Disallow with Google Robots for Crawl your Post, Pages etc. You set it as your requirement after that Google Robots crawl your post pages you have set up in Robots Txt File.

SanGoku · 03-22-2016, 08:31 PM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

hap53144 · 03-23-2016, 08:58 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
Thanks

03-18-2016, 01:21 AM	#1
europeanoversea Registered User Join Date: Jan 2016 Posts: 24	what is the use of robot.txt? what is the use of robot.txt? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-18-2016, 01:52 AM	#2
alicemark15 Registered User Join Date: Oct 2015 Posts: 95	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-18-2016, 01:58 AM	#3
arianagrand Registered User Join Date: Feb 2014 Posts: 197	robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-18-2016, 03:00 AM	#4
daviddakarai Registered User Join Date: Jan 2015 Posts: 677	Robots.txt is a simple text file that is placed on your site’s root directory. It is that file on your website that tells these search engine robots what to crawl and what not to crawl on your site. To facilitate auto-discovery of your sitemap file through your robots.txt, all you have to do is place a directive with the URL in your robots.txt __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-18-2016, 03:17 AM	#5
krify Registered User Join Date: Feb 2016 Posts: 83	The main use of Robot text file is to make search engines know what is are the URLs need to cowl around the websites ! robot dot text is used to upload at the time of site creation then site with sitemap upload using ftp publications. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. KRIFY DOT CO

03-18-2016, 06:00 AM	#6
AshokDixit89 Registered User Join Date: Jan 2016 Location: Mumbai, India Posts: 1,064	Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information. Robots.txt Allowing Access to Specific Crawlers. Allow everything apart from certain patterns of URLs. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-18-2016, 11:20 PM	#7
indianlawwatch Registered User Join Date: Feb 2016 Posts: 44	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-19-2016, 01:24 AM	#8
Sunsite Registered User Join Date: Mar 2016 Posts: 8	It is a instruction to robots in a text how to crawl and index pages on their website __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. business online To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-19-2016, 04:14 AM	#9
Sangeetakashyap Registered User Join Date: Sep 2015 Location: Delhi Posts: 145	Robot.txt is a text file that has been used by website owners to give instructions to the web robots about the website. It helps the robots of Google to crawl each page of website so that it can index it properly & can rank it well in the Google search engine easily. It is an important factor of On Page of any website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-22-2016, 04:37 AM	#10
accutranskpo Registered User Join Date: Mar 2016 Posts: 15	robots.txt is a text file which is used by web-master to give crawling instructions to the search engine crawlers which page is to be crawl and which page is not.

03-22-2016, 04:54 AM	#11
vashikaransutra Registered User Join Date: Aug 2015 Posts: 279	It is must file for every website if webmasters want to do SEO. This file tells to Robots about which files should crawl or which not. It must place on root of the website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-22-2016, 05:49 AM	#12
Nexevo Technolo Registered User Join Date: Sep 2015 Location: Bangalore Posts: 548	When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-22-2016, 04:40 PM	#13
aajokhio Registered User Join Date: May 2015 Location: Hyderabad/Pakistan Posts: 25	Basically, Robots Txt File Allow or Disallow with Google Robots for Crawl your Post, Pages etc. You set it as your requirement after that Google Robots crawl your post pages you have set up in Robots Txt File.

03-22-2016, 08:31 PM	#14
SanGoku Registered User Join Date: Apr 2013 Posts: 382	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

03-23-2016, 08:58 AM	#15
hap53144 Registered User Join Date: Feb 2016 Posts: 34	Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results. Thanks __________________ Create Any Stunning Animated Graphics with To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. In Just A Few Clicks \| New trick works in To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. for ANY Facebook Ad \| Turns Your Videos Into Sales Machine at To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools
Show Printable Version Email this Page
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread:

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
what is robot.txt?	rubysethi	Search Engine Optimization	0	12-19-2014 11:27 PM
How to know about robot.txt ?	vanaja	Search Engine Optimization	10	10-08-2014 04:39 AM
how can Create the Robot.txt	pankaj	Search Engine Optimization	3	10-08-2014 03:43 AM
how can create the Robot.txt file	eagleeye dental	Search Engine Optimization	0	09-01-2014 09:40 PM