What is robots.txt used for? - Site Owners Forums

cearsinfotech · 07-15-2016, 05:44 AM

What is robots.txt used for?

tftus · 07-15-2016, 09:15 AM

robots.txt file is used for control search engine bot.

Pillars · 07-15-2016, 11:32 PM

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

buy pin likes · 07-16-2016, 03:41 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

sashwatmegh · 07-16-2016, 03:59 AM

Robots.txt file controls how search engine spiders see and interact with your web pages.

shaunblake12 · 07-19-2016, 05:51 AM

Robots.txt is a file, that instruct to the crawler what pages are to be crawl and index it in to the server

JakMax · 07-19-2016, 06:02 AM

The robots.txt file is allow to give instructions about site to web robots. One can control his page indexing by this robots.txt. It's very important.

RahulKumar · 07-20-2016, 12:48 AM

Robot.txt is a text file that is uploaded to a website's root directory and linked in the html code of the website. A simple file that contains components used to specify the pages on a website that must be crawled by search engine bots.

Pillars · 07-20-2016, 01:49 AM

Robots.txt is a text file that is inserted into your website and contains information for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling.

Danielhendry · 07-20-2016, 01:49 AM

Website communication with web crawler and other web robots. this is basic information for Robot.txt

Thoughtgrid · 07-20-2016, 01:57 AM

The robot.txt is one of the on page optimization technique it give the permission to search engine crawler whether allow or disallow spider index those pages so uses of robot.txt file it give protection and give security to your secret pages

BryceBrady · 11-16-2016, 11:02 PM

Robot.txt is a text file that is uploaded to a website's root directory and linked in the html code of the website. A simple file that contains components used to specify the pages on a website that must be crawled by search engine bots.

sudeepkhana · 11-17-2016, 01:42 AM

Robots.txt is a text (not HTML) file you put on your site to tell search robots which pages you would like them not to visit. The robots.txt is a very simple text file that is placed in your root directory. It creates to instruct the robots (typically search engine robots) how to crawl and index pages on their website.

helennguyen3687 · 11-17-2016, 01:47 AM

robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on website.

janpaul7890 · 11-17-2016, 02:05 AM

robots.txt is a simple text file webmaster create to instruct search engine robots how to crawl and indexed of their website

07-15-2016, 05:44 AM	#1
cearsinfotech Registered User Join Date: Feb 2016 Posts: 72	What is robots.txt used for? What is robots.txt used for? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. ,

07-15-2016, 09:15 AM	#2
tftus Registered User Join Date: Jun 2016 Posts: 3	robots.txt file is used for control search engine bot. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-15-2016, 11:32 PM	#3
Pillars Registered User Join Date: Aug 2015 Location: Chennai, TN, India Posts: 362	Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-16-2016, 03:41 AM	#4
buy pin likes Registered User Join Date: Jul 2016 Posts: 80	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: " means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \|\| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \|\| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \|\| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

07-16-2016, 03:59 AM	#5
sashwatmegh Registered User Join Date: Apr 2016 Location: pune Posts: 583	Robots.txt file controls how search engine spiders see and interact with your web pages. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-19-2016, 05:51 AM	#6
shaunblake12 Registered User Join Date: Jul 2016 Posts: 89	Robots.txt is a file, that instruct to the crawler what pages are to be crawl and index it in to the server __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-19-2016, 06:02 AM	#7
JakMax Registered User Join Date: Jul 2016 Location: Singapore Posts: 31	The robots.txt file is allow to give instructions about site to web robots. One can control his page indexing by this robots.txt. It's very important.

07-20-2016, 12:48 AM	#8
RahulKumar Registered User Join Date: Jun 2016 Posts: 135	Robot.txt is a text file that is uploaded to a website's root directory and linked in the html code of the website. A simple file that contains components used to specify the pages on a website that must be crawled by search engine bots. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-20-2016, 01:49 AM	#9
Pillars Registered User Join Date: Aug 2015 Location: Chennai, TN, India Posts: 362	Robots.txt is a text file that is inserted into your website and contains information for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-20-2016, 01:49 AM	#10
Danielhendry Registered User Join Date: Jul 2016 Posts: 34	Website communication with web crawler and other web robots. this is basic information for Robot.txt __________________ Tel Aviv Apartments \| Tel Aviv Apartment Rentals \| Holiday Rentals Tel Aviv

07-20-2016, 01:57 AM	#11
Thoughtgrid Registered User Join Date: Jul 2016 Location: Bangalore Posts: 210	The robot.txt is one of the on page optimization technique it give the permission to search engine crawler whether allow or disallow spider index those pages so uses of robot.txt file it give protection and give security to your secret pages __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-16-2016, 11:02 PM	#12
BryceBrady Registered User Join Date: Oct 2016 Posts: 580	Robot.txt is a text file that is uploaded to a website's root directory and linked in the html code of the website. A simple file that contains components used to specify the pages on a website that must be crawled by search engine bots. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-17-2016, 01:42 AM	#13
sudeepkhana Registered User Join Date: Jan 2016 Location: India Posts: 1,258	Robots.txt is a text (not HTML) file you put on your site to tell search robots which pages you would like them not to visit. The robots.txt is a very simple text file that is placed in your root directory. It creates to instruct the robots (typically search engine robots) how to crawl and index pages on their website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-17-2016, 01:47 AM	#14
helennguyen3687 Registered User Join Date: Oct 2016 Location: 12102 5th St Houston, TX 77072, USA Posts: 2	robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-17-2016, 02:05 AM	#15
janpaul7890 Registered User Join Date: Jul 2016 Location: USA Posts: 138	robots.txt is a simple text file webmaster create to instruct search engine robots how to crawl and indexed of their website __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM