Go Back   Site Owners Forums - Webmaster Forums > Search Engine Optimization > Search Engine Optimization

Notices


Reply
 
Thread Tools Rate Thread Display Modes
Old 12-24-2016, 04:45 AM   #1
Reshma123
Registered User
 
Join Date: Nov 2016
Posts: 53
What is robots.txt?

What is robots.txt?
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Reshma123 is offline   Reply With Quote

Old 12-24-2016, 05:14 AM   #2
jeniasir
Registered User
 
Join Date: Dec 2016
Posts: 34
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
jeniasir is offline   Reply With Quote
Old 12-25-2016, 11:13 PM   #3
tennywilson
Registered User
 
Join Date: Nov 2016
Posts: 356
A robots.txt file is a text file, following a strict syntax. It’s going to be read by search engine spiders. These spiders are also called robots, hence the name. The syntax is strict simply because it has to be computer readable. There’s no reading between the lines here, something is either 1, or 0.
Also called the “Robots Exclusion Protocol”, the robots.txt file is the result of a consensus between early search engine spider developers. It’s not an official standard by any standards organization, but all major search engines do adhere to it.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
tennywilson is offline   Reply With Quote
Old 12-25-2016, 11:14 PM   #4
collaberex
Registered User
 
Join Date: Oct 2016
Posts: 335
Robots exclusion protocol is a standard used by the websites to communicate with the web crawlers and other web robots
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
collaberex is offline   Reply With Quote
Old 12-25-2016, 11:16 PM   #5
Ajaysharma
Registered User
 
Join Date: Nov 2016
Posts: 372
Robots.txt is a text file. It is through this file, it gives instruction to search engine crawlers about indexing and caching of a webpage, file of a website or directory, domain.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Ajaysharma is offline   Reply With Quote
Old 12-26-2016, 01:42 AM   #6
DeanAlbert123
Registered User
 
Join Date: Aug 2016
Location: USA
Posts: 84
It is a type of text file through which we tells the bots what to crawl or what to not
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
||
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
||
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
DeanAlbert123 is offline   Reply With Quote
Old 12-26-2016, 01:45 AM   #7
Goldysoni
Registered User
 
Join Date: Dec 2016
Posts: 66
robot.txt is a text file created in webmaster tool that instructs the crawler to index.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Goldysoni is offline   Reply With Quote
Old 12-26-2016, 01:52 AM   #8
marktyler
Registered User
 
Join Date: Dec 2016
Posts: 229
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
marktyler is offline   Reply With Quote
Old 12-26-2016, 03:47 AM   #9
Aquawaterjet
Registered User
 
Join Date: Dec 2016
Posts: 109
Robots.txt is a text file used to give instructions to the search engine crawlers about the caching and indexing of a webpage, domain, directory or a file of a website.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Aquawaterjet is offline   Reply With Quote
Old 12-26-2016, 04:41 AM   #10
Jackmack
Registered User
 
Join Date: Dec 2016
Posts: 32
Robots.txt is a text file.
it gives instruction to search engine crawlers about indexing and caching of a webpage, file of a website or directory, domain.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Jackmack is offline   Reply With Quote
Old 12-26-2016, 04:49 AM   #11
itabhyas
Registered User
 
Join Date: Nov 2016
Location: hyderabad
Posts: 154
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter.

Business Analyst Qlikview SAS Tableau Testing tools
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
itabhyas is offline   Reply With Quote
Old 12-26-2016, 05:30 AM   #12
jackwilson
Registered User
 
Join Date: Nov 2016
Posts: 74
robots.txt is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
jackwilson is offline   Reply With Quote
Old 12-26-2016, 06:29 AM   #13
marynancy
Registered User
 
Join Date: Dec 2016
Posts: 66
The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned....
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
marynancy is offline   Reply With Quote
Old 12-26-2016, 07:48 AM   #14
luffy268
Registered User
 
Join Date: Dec 2016
Posts: 94
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
luffy268 is offline   Reply With Quote
Old 12-27-2016, 12:40 AM   #15
Jayasree
Registered User
 
Join Date: Oct 2016
Posts: 252
Robots.txt is a text file.
It instruction to search engine crawlers about indexing and caching of a webpage, file of a website or directory, domain.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Jayasree is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Problem with robots.txt file basrtaj Search Engine Optimization 9 12-31-2016 07:07 AM
Robots.txt jaysh4922 Search Engine Optimization 13 07-28-2016 05:10 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt Fking General Discussion 1 12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress? geniusoptimizer PHP / mySQL 1 04-12-2013 12:36 PM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 02:19 AM


All times are GMT -7. The time now is 11:51 AM.


Powered by vBulletin Copyright © 2020 vBulletin Solutions, Inc.