What is the use of Robots.txt..? - Site Owners Forums

AshokDixit89 · 06-02-2016, 01:05 AM

What is the use of Robots.txt..?

vienchaua10 · 06-02-2016, 02:08 AM

"Robots.txt" redirects here. For Wikipedia's robots.txt files, see the MediaWiki Robots.txt file and the English Wikipedia Robots.txt file.
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize web sites. Not all robots cooperate with the standard; email harvesters, spambots, malware and Search Engines robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard is different from, but can be used in conjunction with Sitemaps, a robot inclusion standard for websites.

SEONinja · 06-02-2016, 04:32 AM

Robot.txt is a text file. It gives instruction to search engine crawlers about indexing and caching of a web page, file of a website or directory, domain.

SaraSanjay · 06-02-2016, 04:44 AM

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

yasmineassif · 06-02-2016, 05:33 AM

Website owners use the /robots.txt file to give instructions about their website to robots. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

pavandarsigunta · 06-02-2016, 06:01 AM

it will guide google spiders to crawl your site properly. it will tell to the spiders not to verify specific folder or file in your blog.

priyankayadav · 06-02-2016, 06:09 AM

Utilizing a robots.txt document is simple, yet requires access to your server's root area. Case in point, if your site is situated at:

http://adomain.com/mysite/index.html

you should have the capacity to make a record situated here:

http://adomain.com/robots.txt

RahulKumar · 06-02-2016, 10:40 PM

Robots.txt is a text file that is uploaded to a website's root directory and linked in the html code of the website. The robots.txt is used to provide instructions about the website to web robots and spiders.

Pillars · 06-03-2016, 02:05 AM

Robots.txt is a text file that is inserted into your website and contains information for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling.

06-02-2016, 01:05 AM	#1
AshokDixit89 Registered User Join Date: Jan 2016 Location: Mumbai, India Posts: 1,064	What is the use of Robots.txt..? What is the use of Robots.txt..? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-02-2016, 02:08 AM	#2
vienchaua10 Registered User Join Date: May 2016 Posts: 36	"Robots.txt" redirects here. For Wikipedia's robots.txt files, see the MediaWiki Robots.txt file and the English Wikipedia Robots.txt file. The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize web sites. Not all robots cooperate with the standard; email harvesters, spambots, malware and Search Engines robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard is different from, but can be used in conjunction with Sitemaps, a robot inclusion standard for websites. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - – It is a launch platform which allows you to create a JV page

06-02-2016, 04:32 AM	#3
SEONinja Registered User Join Date: May 2016 Location: Florida, United State Posts: 36	Robot.txt is a text file. It gives instruction to search engine crawlers about indexing and caching of a web page, file of a website or directory, domain. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-02-2016, 04:44 AM	#4
SaraSanjay Registered User Join Date: May 2016 Posts: 551	When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-02-2016, 06:01 AM	#6
pavandarsigunta Registered User Join Date: May 2016 Posts: 14	it will guide google spiders to crawl your site properly. it will tell to the spiders not to verify specific folder or file in your blog. __________________ pawan kumar SEO guide To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-02-2016, 05:33 AM	#5
yasmineassif Registered User Join Date: Jun 2016 Location: lebanon Posts: 8	Website owners use the /robots.txt file to give instructions about their website to robots. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

06-02-2016, 06:09 AM	#7
priyankayadav Registered User Join Date: May 2016 Posts: 60	Utilizing a robots.txt document is simple, yet requires access to your server's root area. Case in point, if your site is situated at: http://adomain.com/mysite/index.html you should have the capacity to make a record situated here: http://adomain.com/robots.txt __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-02-2016, 10:40 PM	#8
RahulKumar Registered User Join Date: Jun 2016 Posts: 135	Robots.txt is a text file that is uploaded to a website's root directory and linked in the html code of the website. The robots.txt is used to provide instructions about the website to web robots and spiders. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-03-2016, 02:05 AM	#9
Pillars Registered User Join Date: Aug 2015 Location: Chennai, TN, India Posts: 362	Robots.txt is a text file that is inserted into your website and contains information for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM