what is robots.txt? - Page 2 - Site Owners Forums

sirishasiri · 12-06-2017, 06:18 AM

Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.

maheshseo · 12-06-2017, 06:19 AM

robots.txt is mainly used to allow google crawlers for indexing purpose

ROYCPO · 12-06-2017, 06:45 AM

Website owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol

oyecaptain · 12-06-2017, 11:14 PM

Robots txt in generally for use information our crawlers website.

leenahahim · 12-08-2017, 09:48 PM

I helps to block all web crawler form all webpages or specific web crawler form specif webpage.

samedaytoursin · 12-09-2017, 02:04 AM

Robot txt is the exclusion which is used the proper scanned files.

zahrafarida · 12-09-2017, 01:07 PM

Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.

nathanleo · 11-26-2021, 10:23 PM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.

12-06-2017, 06:19 AM	#17
maheshseo Registered User Join Date: Jun 2017 Posts: 23	robots.txt is mainly used to allow google crawlers for indexing purpose __________________ German Language Classes in Bangalore\|French Classes in Bangalore\|English Courses in Bangalore

12-06-2017, 06:45 AM	#18
ROYCPO Registered User Join Date: Dec 2015 Location: Bangladesh Posts: 267	Website owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol __________________ Clipping Path Outsource, digital graphics studio providing image editing services all over the world. Expert in clipping path Service.

12-08-2017, 09:48 PM	#20
leenahahim Registered User Join Date: Feb 2016 Posts: 459	I helps to block all web crawler form all webpages or specific web crawler form specif webpage. __________________ Movers in Abu Dhabi \| Best Movers in UAE \| Best Movers in Abu Dhabi \| Best Movers in Dubai

12-09-2017, 02:04 AM	#21
samedaytoursin Registered User Join Date: Oct 2015 Location: Agra Posts: 232	Robot txt is the exclusion which is used the proper scanned files. __________________ Taj Mahal Tours Tickets \| Taj Mahal Tours \| Delhi Agra Jaipur Tour \| Golden Triangle Tour 4 Days \| Agra Tour Packages \| Delhi Tour Packages \| Sunrise Taj Mahal Tour \| Mathura Vrindavan Tour \| Taj Mahal Tour by Car \| About Taj Mahal \| Jaipur Tours

12-09-2017, 01:07 PM	#22
zahrafarida Registered User Join Date: Dec 2017 Location: hydarabad Posts: 37	Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. __________________ Professional Scrum Master training \| Professional Scrum Foundation Training \| Certified less practitioner training

12-06-2017, 06:18 AM	#16
sirishasiri Registered User Join Date: Dec 2017 Posts: 57	Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.

12-06-2017, 11:14 PM	#19
oyecaptain Registered User Join Date: Nov 2016 Location: India Posts: 81	Robots txt in generally for use information our crawlers website.

11-26-2021, 10:23 PM	#23
nathanleo Registered User Join Date: Jul 2017 Posts: 2,283	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. __________________ Villa Plots in Chennai Plots Sale in Kelambakkam Premium Villa Plots in Kelambakkam fastest selling villa plots in chennai Villa Plots in Kelambakkam OMR Residential Plots for Sale in OMR Premium Villa Plots in OMR Villa Plots in Kelambakkam Villa Plots Sale in Kelambakkam Plots Sale in Kelambakkam Residential Plots in Kelambakkam Land for Sale Near Chennai Plots for Sale in Chennai Premium Villa Plots OMR luxury villa plots in ECR Chennai

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Robots.txt or Meta Robots.txt???	santoshkumar	Search Engine Optimization	10	02-14-2017 04:03 AM
Robots.txt	jaysh4922	Search Engine Optimization	13	07-28-2016 05:10 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM