Site Owners Forums - Webmaster Forums

Site Owners Forums - Webmaster Forums (http://siteownersforums.com/index.php)
-   Search Engine Optimization (http://siteownersforums.com/forumdisplay.php?f=16)
-   -   What is robots.txt? (http://siteownersforums.com/showthread.php?t=191107)

kumarvinod 03-19-2017 11:07 PM

What is robots.txt?
 
Hlo Friends,
Can anyone tell me, What is robots.txt?

sharmaroshni012 03-19-2017 11:10 PM

A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want to be accessed by search engine crawlers. The file uses the Robots Exclusion Standard, which is a protocol with a small set of commands that can be used to indicate access to your site by section and by specific kinds of web crawlers (such as mobile crawlers vs desktop crawlers).

friendhrm 03-19-2017 11:15 PM

A robots.txt is a small file that tells a search engine which pages to index and which pages to ignore.

A robots.txt is written to tell the 'bots which pages, or parts of the websites that should be indexed and which parts should not. A 'good' robot will follow these instructions.

sonvi.belani 03-19-2017 11:32 PM

Hi,
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way. So, if you don't put robots.txt in the right place, do not be surprised that search engines index your whole site.
Thanks

shakthipriya 03-19-2017 11:35 PM

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door. That is why we say that if you have really sensitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

cinemagicllc 03-19-2017 11:44 PM

Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

jaychristopher 03-19-2017 11:46 PM

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.

tennywilson 03-19-2017 11:57 PM

The Robots.txt file of a website will work when it is used as a request to specific robots to ignore directories or files specified within the Robots.txt file.

Ajaysharma 03-20-2017 01:39 AM

Robots.txt is a file in the root directory of your web site that instructs web crawlers what parts, or all, or none of your site they are allowed examine.

emilyoliver 03-20-2017 03:30 AM

The Robots.txt is the file used to restrict the crawler from crawl the secured files.

RH-Calvin 03-21-2017 10:33 AM

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

jasonroy21 03-21-2017 10:05 PM

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site.

Atman 03-21-2017 10:09 PM

The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

darrensmith67 03-22-2017 10:00 PM

Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.

Justsee 03-22-2017 10:04 PM

Robots.txt it's a kind of text file, it provide instruction to the crawler about caching, indexing of a website.


All times are GMT -7. The time now is 01:36 AM.


Powered by vBulletin Copyright © 2020 vBulletin Solutions, Inc.