Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > Internet Promotion > Web Promotion

Notices


Reply
 
Thread Tools Rating: Thread Rating: 7 votes, 2.57 average. Display Modes
Old 09-07-2016, 04:46 AM   #1
bellevoir
Registered User
 
Join Date: Jan 2016
Location: 3579 E. Foothill Blvd. #190 Pasadena, CA 91107-3119
Posts: 74
What is the use of robots.txt?

What is the use of robots.txt?
bellevoir is offline   Reply With Quote

Old 09-07-2016, 09:32 AM   #2
webdesigndubai
Registered User
 
Join Date: Sep 2016
Location: Dubai, UAE
Posts: 2
Robot.txt file is use to disallow pages which having confidential information
__________________
Web Design Dubai Company
webdesigndubai is offline   Reply With Quote
Old 09-07-2016, 11:33 PM   #3
alishasmith
Registered User
 
Join Date: Aug 2016
Location: Ghaziabad
Posts: 52
Robots.txt is a text file which gives the instructions to crawlers about the website that which pages should be crawled or not.
alishasmith is offline   Reply With Quote
Old 09-08-2016, 02:52 AM   #4
Pillars
Registered User
 
Join Date: Aug 2015
Posts: 350
Robots.txt is a text file that is inserted into your website and contains information for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling.
__________________
Fashion | Entertainment | Lifestyle | Pets | Food
Pillars is offline   Reply With Quote
Old 09-08-2016, 02:57 AM   #5
illuminatingasi
Registered User
 
Join Date: Aug 2015
Posts: 92
Robots.txt file allows to search engine spider which one page of website will be cache.
illuminatingasi is offline   Reply With Quote
Old 09-12-2016, 11:05 PM   #6
RH-Calvin
Registered User
 
Join Date: May 2015
Location: Forum
Posts: 2,012
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
__________________
Cheap VPS | $1 VPS Hosting
Cheap Windows VPS | Windows Server with Remote Desktop Access
Cheap Dedicated Server | Servers with Free IPMI Setup
RH-Calvin is offline   Reply With Quote
Old 09-13-2016, 02:31 AM   #7
cedric903
Registered User
 
Join Date: Jul 2016
Posts: 36
Thanks, it very useful.
cedric903 is offline   Reply With Quote
Old 09-13-2016, 04:32 AM   #8
George re
Registered User
 
Join Date: Sep 2016
Posts: 36
Robots.txt is a text file which gives the instructions to crawlers about the website that which pages should be crawled or not.
George re is offline   Reply With Quote
Old 09-14-2016, 03:14 AM   #9
jasonroy21
Registered User
 
Join Date: Jul 2016
Posts: 864
Robots.txt is a text file which gives the instructions to crawlers about the website that will not allow to visit
jasonroy21 is offline   Reply With Quote
Old 09-19-2016, 12:59 AM   #10
AshokDixit89
Registered User
 
Join Date: Jan 2016
Location: Mumbai, India
Posts: 1,161
Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.
AshokDixit89 is offline   Reply With Quote
Old 09-19-2016, 05:11 AM   #11
lisaryan
Registered User
 
Join Date: Sep 2016
Location: Bangalore
Posts: 64
The robots.txt file helps search engines understand which pages on a website should be indexed and crawled. This file can also be used to block the unwanted crawlers to search your websites.
lisaryan is offline   Reply With Quote
Old 07-06-2017, 11:49 PM   #12
pxljobs
Registered User
 
Join Date: Jul 2016
Location: Bangalore
Posts: 431
These file give permission to search engine crawler which one to access and which one to not access these are helpful to improve the security.
pxljobs is offline   Reply With Quote
Old 07-07-2017, 03:01 AM   #13
alexson
Registered User
 
Join Date: Jan 2017
Location: Arizona
Posts: 163
Through robots.txt you can stop search bots to crawl your websites. You can also use meta robots to prevent your website or web page.
alexson is offline   Reply With Quote
Old 07-07-2017, 03:12 AM   #14
foodmachinex
Registered User
 
Join Date: Jun 2017
Posts: 302
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
foodmachinex is offline   Reply With Quote
Old 07-07-2017, 03:16 AM   #15
newproaudio
Registered User
 
Join Date: May 2017
Posts: 365
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
newproaudio is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt Fking General Discussion 2 09-20-2018 03:44 AM
Robots.txt jaysh4922 Search Engine Optimization 15 07-29-2016 05:07 AM
Crawlers, spider & robots.txt? jackthomas087 Search Engine Optimization 0 04-22-2014 03:29 AM
How to edit virtual robots.txt file in wordpress? geniusoptimizer PHP / mySQL 1 04-12-2013 12:36 PM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 02:19 AM


All times are GMT -7. The time now is 03:54 PM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.