What is robots.txt? - Site Owners Forums

santoshkumar · 01-03-2017, 04:48 AM

What is robots.txt?

FerinKings · 01-03-2017, 04:51 AM

robots.txt is a text file webmasters create to instruct robots, how to crawl and index pages on their website.

LukeBarter · 01-03-2017, 05:09 AM

It is a text file, create to instruct robots,how to crawl and index pages on their website.

Carlbrewster · 01-03-2017, 05:21 AM

robots.txt a file used to put validation on a specific file or folder.

puneet3214 · 01-03-2017, 10:44 AM

Robots.txt is a text file where we write a simple code to google crawl or not.

Gowtham20 · 01-03-2017, 10:25 PM

Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.

jeffronald19 · 01-03-2017, 10:37 PM

Robots.txt is a standard used by websites to communicate with web crawlers and other web robots.

rajj · 01-03-2017, 10:42 PM

Robots.txt is a text file, and it tells the google crawl or not to crawl particular file or folder.

alexis jordan · 01-03-2017, 10:48 PM

The robots exclusion customary, conjointly referred to as the robots exclusion protocol or just robots.txt, could be a customary employed by netsites to speak with net crawlers and different web robots. the quality specifies the way to inform the net automaton regarding that areas of the web site mustn't be processed or scanned.

Click here for more details about how to bring users back to your app

Gracechloe123 · 01-03-2017, 10:49 PM

Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders.

mjohn271 · 01-03-2017, 11:17 PM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

rajkiran · 01-04-2017, 02:17 AM

robots.txt is a text file webmasters create to instruct robots, how to crawl and index pages on their website.

tennywilson · 01-04-2017, 02:29 AM

Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.

stuartsmithh5 · 01-04-2017, 02:40 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize web sites.

saurabh3214 · 01-04-2017, 02:48 AM

Robots.txt is text file where we can allow or disallow webpage.

01-03-2017, 04:48 AM	#1
santoshkumar Registered User Join Date: Oct 2016 Posts: 30	What is robots.txt? What is robots.txt?

01-03-2017, 04:51 AM	#2
FerinKings Registered User Join Date: Aug 2016 Posts: 205	robots.txt is a text file webmasters create to instruct robots, how to crawl and index pages on their website. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

01-03-2017, 05:09 AM	#3
LukeBarter Registered User Join Date: Sep 2016 Posts: 171	It is a text file, create to instruct robots,how to crawl and index pages on their website. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \|\| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

01-03-2017, 05:21 AM	#4
Carlbrewster Registered User Join Date: Aug 2016 Posts: 140	robots.txt a file used to put validation on a specific file or folder. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-03-2017, 10:25 PM	#6
Gowtham20 Registered User Join Date: Nov 2016 Posts: 162	*Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-03-2017, 10:44 AM	#5
puneet3214 Registered User Join Date: Dec 2016 Location: Delhi Posts: 16	Robots.txt is a text file where we write a simple code to google crawl or not.

01-03-2017, 10:37 PM	#7
jeffronald19 Registered User Join Date: Nov 2016 Posts: 127	Robots.txt is a standard used by websites to communicate with web crawlers and other web robots. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-03-2017, 10:42 PM	#8
rajj Registered User Join Date: Dec 2016 Posts: 36	Robots.txt is a text file, and it tells the google crawl or not to crawl particular file or folder. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-03-2017, 10:48 PM	#9
alexis jordan Registered User Join Date: Nov 2016 Location: USA Posts: 32	The robots exclusion customary, conjointly referred to as the robots exclusion protocol or just robots.txt, could be a customary employed by netsites to speak with net crawlers and different web robots. the quality specifies the way to inform the net automaton regarding that areas of the web site mustn't be processed or scanned. Click here for more details about how to bring users back to your app __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-03-2017, 10:49 PM	#10
Gracechloe123 Registered User Join Date: Dec 2016 Posts: 85	Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-03-2017, 11:17 PM	#11
mjohn271 Registered User Join Date: Aug 2016 Posts: 175	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-04-2017, 02:17 AM	#12
rajkiran Registered User Join Date: Aug 2016 Posts: 961	robots.txt is a text file webmasters create to instruct robots, how to crawl and index pages on their website.

01-04-2017, 02:29 AM	#13
tennywilson Registered User Join Date: Nov 2016 Posts: 356	Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-04-2017, 02:40 AM	#14
stuartsmithh5 Registered User Join Date: Oct 2016 Posts: 81	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize web sites. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-04-2017, 02:48 AM	#15
saurabh3214 Registered User Join Date: Dec 2016 Posts: 6	Robots.txt is text file where we can allow or disallow webpage.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Robots.txt	jaysh4922	Search Engine Optimization	13	07-28-2016 05:10 AM
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM