Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > Search Engine Optimization > Search Engine Optimization

Notices


Reply
 
Thread Tools Rate Thread Display Modes
Old 03-20-2017, 12:07 AM   #1
kumarvinod
Registered User
 
Join Date: Mar 2017
Posts: 52
What is robots.txt?

Hlo Friends,
Can anyone tell me, What is robots.txt?
kumarvinod is offline   Reply With Quote

Old 03-20-2017, 12:10 AM   #2
sharmaroshni012
Registered User
 
Join Date: Nov 2016
Posts: 510
A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want to be accessed by search engine crawlers. The file uses the Robots Exclusion Standard, which is a protocol with a small set of commands that can be used to indicate access to your site by section and by specific kinds of web crawlers (such as mobile crawlers vs desktop crawlers).
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
sharmaroshni012 is offline   Reply With Quote
Old 03-20-2017, 12:15 AM   #3
friendhrm
Registered User
 
Join Date: Feb 2017
Posts: 175
A robots.txt is a small file that tells a search engine which pages to index and which pages to ignore.

A robots.txt is written to tell the 'bots which pages, or parts of the websites that should be indexed and which parts should not. A 'good' robot will follow these instructions.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
friendhrm is offline   Reply With Quote
Old 03-20-2017, 12:32 AM   #4
sonvi.belani
Registered User
 
Join Date: Jan 2017
Location: Ahmedabad
Posts: 126
Hi,
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.

The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way. So, if you don't put robots.txt in the right place, do not be surprised that search engines index your whole site.
Thanks
__________________
Sonvi Belani
Ecommerce Website Developer

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
sonvi.belani is offline   Reply With Quote
Old 03-20-2017, 12:35 AM   #5
shakthipriya
Registered User
 
Join Date: Feb 2017
Posts: 134
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door. That is why we say that if you have really sensitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
shakthipriya is offline   Reply With Quote
Old 03-20-2017, 12:44 AM   #6
cinemagicllc
Registered User
 
Join Date: Oct 2015
Location: New Jersey
Posts: 199
Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
cinemagicllc is offline   Reply With Quote
Old 03-20-2017, 12:46 AM   #7
jaychristopher
Registered User
 
Join Date: Nov 2016
Posts: 271
Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
jaychristopher is offline   Reply With Quote
Old 03-20-2017, 12:57 AM   #8
tennywilson
Registered User
 
Join Date: Nov 2016
Posts: 396
The Robots.txt file of a website will work when it is used as a request to specific robots to ignore directories or files specified within the Robots.txt file.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
tennywilson is offline   Reply With Quote
Old 03-20-2017, 02:39 AM   #9
Ajaysharma
Registered User
 
Join Date: Nov 2016
Posts: 412
Robots.txt is a file in the root directory of your web site that instructs web crawlers what parts, or all, or none of your site they are allowed examine.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Ajaysharma is offline   Reply With Quote
Old 03-20-2017, 04:30 AM   #10
emilyoliver
Registered User
 
Join Date: Sep 2016
Posts: 175
The Robots.txt is the file used to restrict the crawler from crawl the secured files.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
emilyoliver is offline   Reply With Quote
Old 03-21-2017, 11:33 AM   #11
RH-Calvin
Registered User
 
Join Date: May 2015
Location: Forum
Posts: 1,560
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
| $1 VPS Hosting

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
| Windows Server with Remote Desktop Access

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
| Servers with Free IPMI Setup
RH-Calvin is offline   Reply With Quote
Old 03-21-2017, 11:05 PM   #12
jasonroy21
Registered User
 
Join Date: Jul 2016
Posts: 864
Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
jasonroy21 is offline   Reply With Quote
Old 03-21-2017, 11:09 PM   #13
Atman
Registered User
 
Join Date: Mar 2017
Posts: 29
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Atman is offline   Reply With Quote
Old 03-22-2017, 11:00 PM   #14
darrensmith67
Registered User
 
Join Date: Dec 2016
Posts: 415
Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
darrensmith67 is offline   Reply With Quote
Old 03-22-2017, 11:04 PM   #15
Justsee
Registered User
 
Join Date: Mar 2017
Posts: 40
Robots.txt it's a kind of text file, it provide instruction to the crawler about caching, indexing of a website.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Justsee is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Robots.txt or Meta Robots.txt??? santoshkumar Search Engine Optimization 11 02-14-2017 04:03 AM
Robots.txt jaysh4922 Search Engine Optimization 15 07-29-2016 06:07 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt Fking General Discussion 1 12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress? geniusoptimizer PHP / mySQL 1 04-12-2013 01:36 PM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 03:19 AM


All times are GMT -7. The time now is 12:34 PM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.