What is robots.txt? - Page 2 - Site Owners Forums

zitoitalahold · 12-13-2012, 10:05 PM

Robots.txt is a text (not html) files which search robots in your site which pages are not visited. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site.

johnstamos00000 · 12-21-2012, 03:51 AM

The robots.txt is a file that follow the search engine instruction.

martinsherman · 12-26-2012, 11:48 PM

The robots.txt file is a text file that tells search engine crawlers which portions of your website they should NOT index. If you don't want to restrict search engine crawlers, you should simply create an empty robots.txt file (e.g., touch robots.txt) or one that looks like this:

User-agent: *
Disallow:
Once you have created a robots.txt file, you store it in the root directory of your Web server.

Hope this helps you!!

johnstamos00000 · 01-21-2013, 01:50 AM

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

tionnasmith · 07-24-2017, 03:55 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

manalisoni · 07-24-2017, 05:26 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

Jacklincy · 07-24-2017, 06:13 AM

Robots.txt is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable.

wiliamjamesh · 07-25-2017, 02:44 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

Jessyk · 07-26-2017, 04:44 AM

Robot.txt is a simple notepad file through which crawler crawl your website.

leahwilmot · 07-26-2017, 05:15 AM

Robots.txt is a text file which gives instruction to the search engine crawler to which page has to be crawled or not by allowing or disallowing.

quickyadspro · 07-26-2017, 05:19 AM

The robots.txt is a simple text file in your web site that informs search engine bots how to crawl and index website or web pages. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want.

vijayasinterior · 07-26-2017, 09:09 AM

to block unwanted files we will use this record

Chrisdiary · 07-26-2017, 01:47 PM

The importance of adding custom robots.txt to blogger

Blogger comes with a default robots.txt file. However, blogger allows us to customize this file to suit our needs. This is called custom robots.txt file and I will show you the relevance of customizing this file and how to add it to the blogger server.

What is Robots.txt file ?
A robots.txt is a code which instructs web crawlers on how to go about indexing specific pages on search engines. Before web crawlers index web pages, it will first look at this file in order to ascertain the specific instructions required of it to carry out.
So basically, a robots.txt file does two things :
1. It allows search engines to discover contents on the web
2. It allows specific contents to be indexed on search engines and served to those looking for information

A robots.txt file is usually found by default on website or blog host servers like blogger. In this tutorial, I will explain what each line of code of a robots.txt file represents and how to customize it.I will use a blogger's default robots file to illustrate the working principles of the robots.txt.
The code is as shown below:

http://www.chrisdiary.com/2017/07/th...ng-custom.html

nathanleo · 09-19-2022, 06:29 AM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. Robots are often used by search engines to categorize websites.

AzharSEO · 09-21-2022, 11:57 PM

Robots.txt are the files that will tell the crawler which pages of the website to crawl and which pages of the website not to crawl.

12-13-2012, 10:05 PM	#16
zitoitalahold Registered User Join Date: Dec 2012 Posts: 36	Robots.txt is a text (not html) files which search robots in your site which pages are not visited. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. is a famous social media site and it has a dedicated customer care team to deal To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. refund issues.

12-21-2012, 03:51 AM	#17
johnstamos00000 Registered User Join Date: Nov 2012 Posts: 214	The robots.txt is a file that follow the search engine instruction. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

12-26-2012, 11:48 PM	#18
martinsherman Registered User Join Date: Nov 2012 Location: India Posts: 33	The robots.txt file is a text file that tells search engine crawlers which portions of your website they should NOT index. If you don't want to restrict search engine crawlers, you should simply create an empty robots.txt file (e.g., touch robots.txt) or one that looks like this: User-agent: * Disallow: Once you have created a robots.txt file, you store it in the root directory of your Web server. Hope this helps you!! __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-21-2013, 01:50 AM	#19
johnstamos00000 Registered User Join Date: Nov 2012 Posts: 214	Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

07-24-2017, 03:55 AM	#20
tionnasmith Registered User Join Date: Sep 2016 Posts: 226	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-24-2017, 05:26 AM	#21
manalisoni Registered User Join Date: Jul 2017 Posts: 258	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-24-2017, 06:13 AM	#22
Jacklincy Registered User Join Date: Dec 2016 Posts: 113	Robots.txt is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-25-2017, 02:44 AM	#23
wiliamjamesh Registered User Join Date: Sep 2016 Posts: 617	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-26-2017, 04:44 AM	#24
Jessyk Registered User Join Date: Jul 2017 Posts: 28	Robot.txt is a simple notepad file through which crawler crawl your website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-26-2017, 05:15 AM	#25
leahwilmot Registered User Join Date: Oct 2016 Posts: 29	Robots.txt is a text file which gives instruction to the search engine crawler to which page has to be crawled or not by allowing or disallowing. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-26-2017, 05:19 AM	#26
quickyadspro Registered User Join Date: Jul 2017 Posts: 70	The robots.txt is a simple text file in your web site that informs search engine bots how to crawl and index website or web pages. It is great when search engines frequently visit your site and index your content but often there are cases when indexing parts of your online content is not what you want. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

07-26-2017, 09:09 AM	#27
vijayasinterior Registered User Join Date: Jul 2017 Posts: 1	to block unwanted files we will use this record __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-26-2017, 01:47 PM	#28
Chrisdiary Registered User Join Date: Jul 2017 Posts: 3	The importance of adding custom robots.txt to blogger Blogger comes with a default robots.txt file. However, blogger allows us to customize this file to suit our needs. This is called custom robots.txt file and I will show you the relevance of customizing this file and how to add it to the blogger server. What is Robots.txt file ? A robots.txt is a code which instructs web crawlers on how to go about indexing specific pages on search engines. Before web crawlers index web pages, it will first look at this file in order to ascertain the specific instructions required of it to carry out. So basically, a robots.txt file does two things : 1. It allows search engines to discover contents on the web 2. It allows specific contents to be indexed on search engines and served to those looking for information A robots.txt file is usually found by default on website or blog host servers like blogger. In this tutorial, I will explain what each line of code of a robots.txt file represents and how to customize it.I will use a blogger's default robots file to illustrate the working principles of the robots.txt. The code is as shown below: http://www.chrisdiary.com/2017/07/th...ng-custom.html

09-19-2022, 06:29 AM	#29
nathanleo Registered User Join Date: Jul 2017 Posts: 2,060	The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. Robots are often used by search engines to categorize websites. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

09-21-2022, 11:57 PM	#30
AzharSEO Registered User Join Date: Aug 2014 Location: India Posts: 774	Robots.txt are the files that will tell the crawler which pages of the website to crawl and which pages of the website not to crawl. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
Robots.txt	sameerseo	Search Engine Optimization	26	09-28-2012 04:18 AM
robots.txt	wheelbarrow001	Google	2	07-27-2012 04:10 AM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM
What Is Robots.txt?	samlko	Search Engine Optimization	13	03-09-2012 02:37 PM