what is robots.txt? - Site Owners Forums

kumar1_16 · 05-27-2017, 05:04 AM

what is robots.txt?

robinmym · 05-27-2017, 09:03 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.

http://www.projuktishop.com/

deepakrajput · 05-27-2017, 11:18 AM

Robots.txt is set of instructions help to index the site.

stellajohn · 05-28-2017, 01:59 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

Mouni · 05-28-2017, 09:37 PM

Robots.txt file instructs the search engine crawlers about pages of the sites it has to index.

Dipika · 05-28-2017, 11:10 PM

Robots.txt is text files mainly report to crawler that it is allowed any website directory. The use of robots.txt is to disallow crawlers from visit private folder or content that gives them no extra information.

masonsilas · 05-28-2017, 11:26 PM

The robots exclusion commonplace conjointly called the robots exclusion protocol or just robots.txt, could be a commonplace utilized by internet sites to speak with internet crawlers and different web robots. the quality specifies the way to inform the net automaton concerning that area of the website shouldn't be processed or scanned.

kingsbridgemast · 05-28-2017, 11:35 PM

Robot.txt file which giveinstruction to Google and other web crawler that whta to crawl and what to not crawl.

Jayasree · 07-21-2017, 01:31 AM

Robots.txt file instructs the search engine crawlers about pages of the sites it has to index.

jenerous · 07-21-2017, 01:59 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

johnkevin · 07-21-2017, 03:21 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

BryceBrady · 07-21-2017, 03:36 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

Vincy · 07-21-2017, 03:52 AM

Robots.txt file is a text file that can be saved to a website’s server. It determines if and when the search engine crawlers can visit a website’s subpages and include them in their index. In doing this, certain subpages can be excluded from the search results.

tionnasmith · 07-21-2017, 04:52 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

Michealdesouza · 12-06-2017, 06:00 AM

The robots rejection standard, otherwise called the robots avoidance convention or just robots.txt, is a standard utilized by sites to speak with web crawlers and other web robots.

05-27-2017, 05:04 AM	#1
kumar1_16 Registered User Join Date: Mar 2017 Posts: 29	what is robots.txt? what is robots.txt? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

05-27-2017, 11:18 AM	#3
deepakrajput Registered User Join Date: Feb 2015 Location: CA Posts: 733	Robots.txt is set of instructions help to index the site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

05-28-2017, 01:59 AM	#4
stellajohn Registered User Join Date: Aug 2013 Location: bangalore Posts: 95	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. __________________ Top 10 Best Web Hosting Companies in India of 2017- To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

05-28-2017, 11:10 PM	#6
Dipika Registered User Join Date: Oct 2016 Location: Bangalore Posts: 242	Robots.txt is text files mainly report to crawler that it is allowed any website directory. The use of robots.txt is to disallow crawlers from visit private folder or content that gives them no extra information. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

05-28-2017, 11:26 PM	#7
masonsilas Registered User Join Date: May 2017 Posts: 47	The robots exclusion commonplace conjointly called the robots exclusion protocol or just robots.txt, could be a commonplace utilized by internet sites to speak with internet crawlers and different web robots. the quality specifies the way to inform the net automaton concerning that area of the website shouldn't be processed or scanned. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

05-27-2017, 09:03 AM	#2
robinmym Registered User Join Date: Feb 2017 Location: Dhaka Posts: 486	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. http://www.projuktishop.com/

05-28-2017, 09:37 PM	#5
Mouni Registered User Join Date: May 2017 Posts: 66	Robots.txt file instructs the search engine crawlers about pages of the sites it has to index.

05-28-2017, 11:35 PM	#8
kingsbridgemast Registered User Join Date: Apr 2017 Posts: 14	Robot.txt file which giveinstruction to Google and other web crawler that whta to crawl and what to not crawl. __________________ if you are looking to buy a To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. then the Kingsbridge Local is the right choice for you.

07-21-2017, 01:31 AM	#9
Jayasree Registered User Join Date: Oct 2016 Posts: 252	Robots.txt file instructs the search engine crawlers about pages of the sites it has to index. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-21-2017, 01:59 AM	#10
jenerous Registered User Join Date: Jan 2017 Posts: 46	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

07-21-2017, 03:21 AM	#11
johnkevin Registered User Join Date: Oct 2016 Posts: 259	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

07-21-2017, 03:36 AM	#12
BryceBrady Registered User Join Date: Oct 2016 Posts: 580	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-21-2017, 03:52 AM	#13
Vincy Registered User Join Date: Jul 2017 Posts: 182	Robots.txt file is a text file that can be saved to a website’s server. It determines if and when the search engine crawlers can visit a website’s subpages and include them in their index. In doing this, certain subpages can be excluded from the search results. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-21-2017, 04:52 AM	#14
tionnasmith Registered User Join Date: Sep 2016 Posts: 226	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-06-2017, 06:00 AM	#15
Michealdesouza Registered User Join Date: Oct 2017 Posts: 145	The robots rejection standard, otherwise called the robots avoidance convention or just robots.txt, is a standard utilized by sites to speak with web crawlers and other web robots. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Robots.txt or Meta Robots.txt???	santoshkumar	Search Engine Optimization	10	02-14-2017 04:03 AM
Robots.txt	jaysh4922	Search Engine Optimization	13	07-28-2016 05:10 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM