What is the use of Robots.txt? - Site Owners Forums

ankitasharma123 · 06-01-2016, 11:07 PM

Hello friends,

What is the use of Robots.txt?

kunalkumar · 06-01-2016, 11:11 PM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

Swatinigam · 06-01-2016, 11:13 PM

Robots exclusion protocol or robot txt is a text file webmaster create to instruct robot typically search engine robot how to crawl and index pages on their website.

Connection phas · 06-06-2016, 03:31 AM

A robots.txt file offers directions to web robots about the pages the site owner doesn't wish to be 'Crawled'. Case in point, in the event that you didn't need your pictures to be recorded by Google and other web indexes, you'd square them utilizing your robots.txt file.

Kunal · 06-06-2016, 04:24 AM

Robot.txt file is a simple code file placed at your web server which instruct search engines bots to how to crawl and index your blog and website.

vipinkedia · 06-06-2016, 04:40 AM

Despite the fact that the robots.txt document is a critical record in the event that you need to have a decent positioning on web search tools, numerous Web locales don't offer this record. On the off chance that your Web website doesn't have a robots.txt document yet, read on to figure out how to make one. On the off chance that you as of now have a robots.txt record, read our tips to ensure that it doesn't contain blunders.

Sonamsingh · 06-07-2016, 01:29 AM

Robots.txt is a text file that is uploading to the Website's root Directory and linked in the HTML code of the Website. The robots.txt file is used to provide instructions about the website to web robots and spiders.

cybexindia15 · 06-07-2016, 04:44 AM

Robots.txt is a text file and part of SEO on page.

SEONinja · 06-07-2016, 04:52 AM

Robot.txt is a text file.
It gives instruction to search engine crawlers about indexing and caching of a web page, file of a website or directory, domain.

jootavoota · 06-07-2016, 05:08 AM

robots.txt is file which is allow to control the search engine bot to index or deindex the website url .

AshokDixit89 · 06-10-2016, 01:21 AM

Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

ritika.patel · 06-10-2016, 03:53 AM

robot.txt is the text that stop the search engine crawler to crawl the website.

nidhitiwari · 06-10-2016, 06:02 AM

Robot.txt file is a simple text file which is used to tell the search engine bots which page or portion to crawl and which one to ignore.

ngocha2991 · 06-10-2016, 07:37 AM

uppppppppppppppppppppppppppppppppppppppppp

RyanVarnals · 06-10-2016, 12:11 PM

Robots.txt helps include or exclude certain webpages from being crawled by search engines. Also note that there is a way to test your robots.txt file for errors through Google Webmaster Tools. Hope this helps.

06-01-2016, 11:07 PM	#1
ankitasharma123 Registered User Join Date: Aug 2013 Posts: 332	What is the use of Robots.txt? Hello friends, What is the use of Robots.txt? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-01-2016, 11:11 PM	#2
kunalkumar Registered User Join Date: Apr 2016 Posts: 198	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: " means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-06-2016, 04:24 AM	#5
Kunal Registered User Join Date: Feb 2016 Posts: 73	Robot.txt file is a simple code file placed at your web server which instruct search engines bots to how to crawl and index your blog and website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-06-2016, 04:40 AM	#6
vipinkedia Registered User Join Date: Jun 2016 Posts: 131	Despite the fact that the robots.txt document is a critical record in the event that you need to have a decent positioning on web search tools, numerous Web locales don't offer this record. On the off chance that your Web website doesn't have a robots.txt document yet, read on to figure out how to make one. On the off chance that you as of now have a robots.txt record, read our tips to ensure that it doesn't contain blunders. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-07-2016, 01:29 AM	#7
Sonamsingh Registered User Join Date: May 2016 Posts: 115	Robots.txt is a text file that is uploading to the Website's root Directory and linked in the HTML code of the Website. The robots.txt file is used to provide instructions about the website to web robots and spiders. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-01-2016, 11:13 PM	#3
Swatinigam Registered User Join Date: May 2016 Posts: 86	Robots exclusion protocol or robot txt is a text file webmaster create to instruct robot typically search engine robot how to crawl and index pages on their website.

06-06-2016, 03:31 AM	#4
Connection phas Registered User Join Date: Jun 2016 Posts: 4	A robots.txt file offers directions to web robots about the pages the site owner doesn't wish to be 'Crawled'. Case in point, in the event that you didn't need your pictures to be recorded by Google and other web indexes, you'd square them utilizing your robots.txt file.

06-07-2016, 04:44 AM	#8
cybexindia15 Registered User Join Date: Sep 2013 Posts: 411	Robots.txt is a text file and part of SEO on page. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

06-07-2016, 04:52 AM	#9
SEONinja Registered User Join Date: May 2016 Location: Florida, United State Posts: 36	Robot.txt is a text file. It gives instruction to search engine crawlers about indexing and caching of a web page, file of a website or directory, domain. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-07-2016, 05:08 AM	#10
jootavoota Registered User Join Date: Apr 2015 Posts: 43	robots.txt is file which is allow to control the search engine bot to index or deindex the website url . __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-10-2016, 01:21 AM	#11
AshokDixit89 Registered User Join Date: Jan 2016 Location: Mumbai, India Posts: 1,064	Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information. Robots.txt Allowing Access to Specific Crawlers. Allow everything apart from certain patterns of URLs. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-10-2016, 03:53 AM	#12
ritika.patel Registered User Join Date: Apr 2016 Posts: 192	robot.txt is the text that stop the search engine crawler to crawl the website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-10-2016, 06:02 AM	#13
nidhitiwari Registered User Join Date: May 2016 Posts: 163	Robot.txt file is a simple text file which is used to tell the search engine bots which page or portion to crawl and which one to ignore. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-10-2016, 07:37 AM	#14
ngocha2991 Registered User Join Date: Jun 2016 Posts: 32	uppppppppppppppppppppppppppppppppppppppppp

06-10-2016, 12:11 PM	#15
RyanVarnals Registered User Join Date: Jun 2016 Posts: 14	Robots.txt helps include or exclude certain webpages from being crawled by search engines. Also note that there is a way to test your robots.txt file for errors through Google Webmaster Tools. Hope this helps. __________________ Ryan Varnals Web Developer To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM