Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > Search Engine Optimization > Search Engine Optimization

Notices


Reply
 
Thread Tools Rate Thread Display Modes
Old 11-27-2017, 05:26 AM   #1
sivakumar1
Registered User
 
Join Date: Jul 2013
Posts: 934
What is the purpose of robots.txt file?

What is the purpose of robots.txt file?
sivakumar1 is offline   Reply With Quote

Old 11-27-2017, 06:05 AM   #2
infiniumtech
Registered User
 
Join Date: Nov 2017
Location: Surat, Gujarat, India
Posts: 51
Website owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
infiniumtech is offline   Reply With Quote
Old 11-27-2017, 09:11 PM   #3
Daisyallie
Registered User
 
Join Date: Nov 2017
Posts: 418
Website owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
Daisyallie is offline   Reply With Quote
Old 11-27-2017, 10:54 PM   #4
fayeseom
Registered User
 
Join Date: Feb 2016
Location: Bangalore
Posts: 488
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.
fayeseom is offline   Reply With Quote
Old 11-27-2017, 11:01 PM   #5
skbplans
Registered User
 
Join Date: Jun 2017
Posts: 410
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
skbplans is offline   Reply With Quote
Old 11-27-2017, 11:10 PM   #6
davidsmith01
Registered User
 
Join Date: Apr 2016
Location: USA
Posts: 269
Quote:
Originally Posted by sivakumar1 View Post
What is the purpose of robots.txt file?
Robot.txt is the file included in the site which indicate Crawler about which pages of the site will be crawled and which not.
davidsmith01 is offline   Reply With Quote
Old 11-27-2017, 11:19 PM   #7
nisha.shiv
Registered User
 
Join Date: Oct 2017
Posts: 43
robots.txt file gives the instruction to the search engine bot which web link should be crawled and which's not.
nisha.shiv is offline   Reply With Quote
Old 11-27-2017, 11:27 PM   #8
JyotikaSharma
Registered User
 
Join Date: Nov 2017
Posts: 1
You can add those pages to a robots.txt file which you do not want to crawl by the google bot.

Thanks,
Jyotika
birthday cake delivery in Lucknow | Birthday cake and flowers delivery in Lucknow
JyotikaSharma is offline   Reply With Quote
Old 11-28-2017, 02:44 AM   #9
sivapriya
Registered User
 
Join Date: Nov 2017
Posts: 65
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent:" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
sivapriya is offline   Reply With Quote
Old 11-28-2017, 08:50 PM   #10
nathanleo
Registered User
 
Join Date: Jul 2017
Posts: 593
The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
nathanleo is offline   Reply With Quote
Old 11-28-2017, 10:05 PM   #11
skbplans
Registered User
 
Join Date: Jun 2017
Posts: 410
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
skbplans is offline   Reply With Quote
Old 11-28-2017, 10:06 PM   #12
intelligentsia1
Registered User
 
Join Date: Aug 2017
Posts: 266
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
intelligentsia1 is offline   Reply With Quote
Old 11-28-2017, 10:19 PM   #13
fusionplus
Registered User
 
Join Date: Dec 2016
Posts: 280
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
fusionplus is offline   Reply With Quote
Old 11-28-2017, 10:20 PM   #14
fusionplusindia
Registered User
 
Join Date: Sep 2017
Posts: 355
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
fusionplusindia is offline   Reply With Quote
Old 11-28-2017, 10:25 PM   #15
agsenterprises
Registered User
 
Join Date: Jul 2017
Posts: 313
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
agsenterprises is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
What is the purpose of robots.txt file? sahithya Search Engine Optimization 18 02-27-2018 07:54 AM
What is the purpose of using robots.txt file in seo? SunFineHanger Search Engine Optimization 15 02-26-2018 01:29 AM
Apple: a new file system encrypted by nature bolhaw General Discussion 0 06-17-2016 07:25 AM
How to edit virtual robots.txt file in wordpress? geniusoptimizer PHP / mySQL 1 04-12-2013 12:36 PM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 02:19 AM


All times are GMT -7. The time now is 06:43 PM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.