Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > Search Engine Optimization > Search Engine Optimization

Notices


Reply
 
Thread Tools Rate Thread Display Modes
Old 03-18-2016, 02:21 AM   #1
europeanoversea
Registered User
 
Join Date: Jan 2016
Posts: 24
what is the use of robot.txt?

what is the use of robot.txt?
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
europeanoversea is offline   Reply With Quote

Old 03-18-2016, 02:52 AM   #2
alicemark15
Registered User
 
Join Date: Oct 2015
Posts: 96
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
alicemark15 is offline   Reply With Quote
Old 03-18-2016, 02:58 AM   #3
arianagrand
Registered User
 
Join Date: Feb 2014
Posts: 187
robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
arianagrand is offline   Reply With Quote
Old 03-18-2016, 04:00 AM   #4
daviddakarai
Registered User
 
Join Date: Jan 2015
Posts: 682
Robots.txt is a simple text file that is placed on your site’s root directory. It is that file on your website that tells these search engine robots what to crawl and what not to crawl on your site. To facilitate auto-discovery of your sitemap file through your robots.txt, all you have to do is place a directive with the URL in your robots.txt
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
daviddakarai is offline   Reply With Quote
Old 03-18-2016, 04:17 AM   #5
krify
Registered User
 
Join Date: Feb 2016
Posts: 86
The main use of Robot text file is to make search engines know what is are the URLs need to cowl around the websites !

robot dot text is used to upload at the time of site creation then site with sitemap upload using ftp publications.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
KRIFY DOT CO
krify is offline   Reply With Quote
Old 03-18-2016, 07:00 AM   #6
AshokDixit89
Registered User
 
Join Date: Jan 2016
Location: Mumbai, India
Posts: 1,084
Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.
AshokDixit89 is offline   Reply With Quote
Old 03-19-2016, 12:20 AM   #7
indianlawwatch
Registered User
 
Join Date: Feb 2016
Posts: 46
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
indianlawwatch is offline   Reply With Quote
Old 03-19-2016, 02:24 AM   #8
Sunsite
Registered User
 
Join Date: Mar 2016
Posts: 9
It is a instruction to robots in a text how to crawl and index pages on their website
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
business online
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Sunsite is offline   Reply With Quote
Old 03-19-2016, 05:14 AM   #9
Sangeetakashyap
Registered User
 
Join Date: Sep 2015
Location: Delhi
Posts: 149
Robot.txt is a text file that has been used by website owners to give instructions to the web robots about the website. It helps the robots of Google to crawl each page of website so that it can index it properly & can rank it well in the Google search engine easily. It is an important factor of On Page of any website.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.



To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.



To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Sangeetakashyap is offline   Reply With Quote
Old 03-22-2016, 05:37 AM   #10
accutranskpo
Registered User
 
Join Date: Mar 2016
Posts: 15
robots.txt is a text file which is used by web-master to give crawling instructions to the search engine crawlers which page is to be crawl and which page is not.
accutranskpo is offline   Reply With Quote
Old 03-22-2016, 05:54 AM   #11
vashikaransutra
Registered User
 
Join Date: Aug 2015
Posts: 138
It is must file for every website if webmasters want to do SEO. This file tells to Robots about which files should crawl or which not. It must place on root of the website.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
vashikaransutra is offline   Reply With Quote
Old 03-22-2016, 06:49 AM   #12
Nexevo Technolo
Registered User
 
Join Date: Sep 2015
Location: Bangalore
Posts: 557
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Nexevo Technolo is offline   Reply With Quote
Old 03-22-2016, 05:40 PM   #13
aajokhio
Registered User
 
Join Date: May 2015
Location: Hyderabad/Pakistan
Posts: 25
Basically, Robots Txt File Allow or Disallow with Google Robots for Crawl your Post, Pages etc. You set it as your requirement after that Google Robots crawl your post pages you have set up in Robots Txt File.
aajokhio is offline   Reply With Quote
Old 03-22-2016, 09:31 PM   #14
SanGoku
Registered User
 
Join Date: Apr 2013
Posts: 388
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
SanGoku is offline   Reply With Quote
Old 03-23-2016, 09:58 AM   #15
hap53144
Registered User
 
Join Date: Feb 2016
Posts: 36
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
Thanks
__________________
Create Any Stunning Animated Graphics with
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
In Just A Few Clicks | New trick works in
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
for ANY Facebook Ad | Turns Your Videos Into Sales Machine at
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
hap53144 is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
what is robot.txt? rubysethi Search Engine Optimization 0 12-19-2014 11:27 PM
How to know about robot.txt ? vanaja Search Engine Optimization 10 10-08-2014 05:39 AM
how can Create the Robot.txt pankaj Search Engine Optimization 3 10-08-2014 04:43 AM
how can create the Robot.txt file eagleeye dental Search Engine Optimization 0 09-01-2014 10:40 PM


All times are GMT -7. The time now is 02:39 PM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.