What is robots.txt - Site Owners Forums

priyakumari95 · 01-12-2018, 01:20 AM

Hello friends,

What is robots.txt.......?

wellliving · 01-12-2018, 01:22 AM

Hi

robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

https://www.welllivingshop.com/beddi...mattress-pads/

stellasweety · 01-12-2018, 01:52 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

PeterGenesis · 01-12-2018, 06:24 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

AmandaCherry · 01-12-2018, 06:51 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

neelseowork · 01-12-2018, 08:53 PM

Quote:

Originally Posted by priyakumari95

Hello friends,

What is robots.txt.......?

simple searchable question asked here.! well no worries, click to know about robots.txt.

charleswesley · 01-13-2018, 02:21 AM

The robots rejection standard, otherwise called the robots prohibition convention or essentially robots.txt, is a standard utilized by sites to speak with web crawlers and other web robots. The standard determines how to advise the web robot about which zones of the site ought not be prepared or filtered.

manival · 01-13-2018, 02:42 AM

Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.

PeterGenesis · 01-14-2018, 02:21 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

01-12-2018, 01:20 AM	#1
priyakumari95 Registered User Join Date: Jul 2017 Posts: 75	What is robots.txt Hello friends, What is robots.txt.......?

01-12-2018, 01:22 AM	#2
wellliving Registered User Join Date: Dec 2015 Location: USA Posts: 1,044	Hi robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. https://www.welllivingshop.com/beddi...mattress-pads/ __________________ organic bedding \| cotton bedding \| organic cotton bedding \| ocotton bedding sets \| Organic latex pillow

01-12-2018, 01:52 AM	#3
stellasweety Registered User Join Date: Mar 2017 Posts: 887	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ Hospital Management System\|Hospital Information System\|Hospital mangement information sytem \| Hospital Information Management System \| Health Information Management

01-12-2018, 06:24 AM	#4
PeterGenesis Registered User Join Date: Dec 2017 Posts: 298	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ nulledbycherry.com \| nulled wordpress themes \| nulled wordpress plugins \| free udemy courses

01-12-2018, 06:51 AM	#5
AmandaCherry Registered User Join Date: Nov 2017 Posts: 400	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ www.nulledbycherry.com \| nulled wordpress themes \| nulled wordpress plugins \| free udemy courses

01-13-2018, 02:21 AM	#7
charleswesley Registered User Join Date: Nov 2017 Posts: 264	The robots rejection standard, otherwise called the robots prohibition convention or essentially robots.txt, is a standard utilized by sites to speak with web crawlers and other web robots. The standard determines how to advise the web robot about which zones of the site ought not be prepared or filtered. __________________ Samsung Showroom in Chennai \| Samsung Smart Phones \| Dell showroom in Chennai

01-13-2018, 02:42 AM	#8
manival Registered User Join Date: Jul 2017 Posts: 433	Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. __________________ Food industry consultant Bakery Equipment Chocolate machines Biscuit Making Machine

01-14-2018, 02:21 AM	#9
PeterGenesis Registered User Join Date: Dec 2017 Posts: 298	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ nulledbycherry.com \| nulled wordpress themes \| nulled wordpress plugins \| free udemy courses

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Thread Tools
Show Printable Version Email this Page
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread:

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Robots.txt or Meta Robots.txt???	santoshkumar	Search Engine Optimization	11	02-14-2017 03:03 AM
Robots.txt	jaysh4922	Search Engine Optimization	15	07-29-2016 05:07 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 10:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM