What is robots.txt? - Site Owners Forums

praveen007 · 05-22-2014, 02:56 AM

It is a kind of text file utilized to provide the instructions to the crawlers about the caching and indexing of a website, directory, domain or a file of a webpage.

larrylo · 05-22-2014, 03:15 AM

You have covered up the basic definition of what it means

austin11 · 05-22-2014, 03:59 AM

Robots.txt helps to index ourwebsite all pages on google.

AvniShergill · 05-23-2014, 12:55 AM

Robots.txt files are used to inform the crawler which URL to crawl and which to avoid.

rummy26 · 05-23-2014, 02:08 AM

Robots.txt files are used to inform the search engine crawler to which URL to allow search engines and which is to disallow .

iedigcom · 05-23-2014, 03:30 AM

Crawling of website by robots is first step to getting ranked in SearchEngines. The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

DanielGarneau · 05-23-2014, 03:36 AM

By way of example, my website is css and html based, which means that it is not currently built with a CMS such as WordPress' and I use the robot.txt file to excule search engines from reaching the development directories I use to test drive CMS softwares such as WordPress. When my WordPress based site is ready for being seen by the world, I will remove the exclusion line from my robot.txt.

You can find the syntax supported by robot.txt files by doing a google search. They will also tell it where to put it on your site, which is the same directory where your index.html file is. If you use a CMS like WordPress, I believe the software generates the robot.txt file according to the parameters you choose within WordPress, but I'm less familiar with these than with the css and html based web sites.

HarveyParker · 05-23-2014, 03:39 AM

Robot.txt is a text file which you put in your website to tell search engine which pages you would like not to crawl.
For an Example:

User Agest: *

Disallow:/admin/
Disallow:/epe/de
Disallow:/dwsdf/fdedx/

ChristinaCa · 05-24-2014, 01:09 AM

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

sarvodyaseo · 05-24-2014, 01:17 AM

Robots.txt is a text file for gives the intruction of bots. particular search engine bots or all search engine bots which file or folder have cached or not cached.

yugant · 10-13-2014, 06:38 AM

robots.txt is a text file that search engine information allow or not.

Rajumakkar07 · 10-13-2014, 01:33 PM

Its pretty much a file that search engines look at to figure out whether to index your site or not.

ezzholidays · 10-13-2014, 11:31 PM

it is a .txt file, that doesn't crawled by the web spider.

ledtubelight · 10-14-2014, 12:05 AM

thanks for the information........

bestindiatours · 10-14-2014, 12:12 AM

its the text file which with the name robot.txt use to disallow the crawler to crawl the page that we don't need to crawl

05-22-2014, 02:56 AM	#1
praveen007 Registered User Join Date: Mar 2014 Posts: 96	What is robots.txt? It is a kind of text file utilized to provide the instructions to the crawlers about the caching and indexing of a website, directory, domain or a file of a webpage. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. ! To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

05-22-2014, 03:15 AM	#2
larrylo Registered User Join Date: May 2014 Posts: 9	You have covered up the basic definition of what it means __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

05-22-2014, 03:59 AM	#3
austin11 Registered User Join Date: Oct 2013 Posts: 77	Robots.txt helps to index ourwebsite all pages on google. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

05-23-2014, 12:55 AM	#4
AvniShergill Registered User Join Date: May 2014 Posts: 89	Robots.txt files are used to inform the crawler which URL to crawl and which to avoid. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

05-23-2014, 02:08 AM	#5
rummy26 Registered User Join Date: Nov 2010 Location: us Posts: 584	Robots.txt files are used to inform the search engine crawler to which URL to allow search engines and which is to disallow . __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

05-23-2014, 03:30 AM	#6
iedigcom Registered User Join Date: May 2014 Posts: 16	Crawling of website by robots is first step to getting ranked in SearchEngines. The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

05-23-2014, 03:36 AM	#7
DanielGarneau Registered User Join Date: May 2014 Location: Quebec City, Quebec, Canada Posts: 23	By way of example, my website is css and html based, which means that it is not currently built with a CMS such as WordPress' and I use the robot.txt file to excule search engines from reaching the development directories I use to test drive CMS softwares such as WordPress. When my WordPress based site is ready for being seen by the world, I will remove the exclusion line from my robot.txt. You can find the syntax supported by robot.txt files by doing a google search. They will also tell it where to put it on your site, which is the same directory where your index.html file is. If you use a CMS like WordPress, I believe the software generates the robot.txt file according to the parameters you choose within WordPress, but I'm less familiar with these than with the css and html based web sites. __________________ Daniel Garneau To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. -- Knowing and Believing: Torn-apart, yet inseparable allies… Last edited by DanielGarneau; 05-23-2014 at 03:41 AM..

05-23-2014, 03:39 AM	#8
HarveyParker Registered User Join Date: May 2014 Location: Mumbai Posts: 6	Robot.txt is a text file which you put in your website to tell search engine which pages you would like not to crawl. For an Example: User Agest: * Disallow:/admin/ Disallow:/epe/de Disallow:/dwsdf/fdedx/ __________________ Party Planet is one of India’s best Party Planner like To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. & To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. Location.

05-24-2014, 01:09 AM	#9
ChristinaCa Registered User Join Date: Mar 2014 Posts: 210	Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

05-24-2014, 01:17 AM	#10
sarvodyaseo Registered User Join Date: Mar 2014 Posts: 86	Robots.txt is a text file for gives the intruction of bots. particular search engine bots or all search engine bots which file or folder have cached or not cached. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-13-2014, 06:38 AM	#11
yugant Registered User Join Date: Sep 2014 Posts: 228	robots.txt is a text file that search engine information allow or not. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-13-2014, 01:33 PM	#12
Rajumakkar07 Registered User Join Date: Oct 2014 Location: Mississauga Posts: 11	Its pretty much a file that search engines look at to figure out whether to index your site or not. __________________ Knowing is not enough, we must apply. Willing is not enough, we must do. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-13-2014, 11:31 PM	#13
ezzholidays Registered User Join Date: Sep 2014 Location: Bangalore Posts: 20	it is a .txt file, that doesn't crawled by the web spider. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-14-2014, 12:05 AM	#14
ledtubelight Registered User Join Date: Aug 2014 Location: Surat Posts: 847	thanks for the information........ __________________ I am SEO expert and handling SEO for following words To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

10-14-2014, 12:12 AM	#15
bestindiatours Registered User Join Date: Oct 2014 Posts: 5	its the text file which with the name robot.txt use to disallow the crawler to crawl the page that we don't need to crawl

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools
Show Printable Version Email this Page
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread:

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM