Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > Internet Promotion > Web Promotion

Notices


Reply
 
Thread Tools Rating: Thread Rating: 7 votes, 2.57 average. Display Modes
Old 09-07-2016, 05:46 AM   #1
bellevoir
Registered User
 
Join Date: Jan 2016
Location: 3579 E. Foothill Blvd. #190 Pasadena, CA 91107-3119
Posts: 72
What is the use of robots.txt?

What is the use of robots.txt?
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
bellevoir is offline   Reply With Quote

Old 09-07-2016, 10:32 AM   #2
webdesigndubai
Registered User
 
Join Date: Sep 2016
Location: Dubai, UAE
Posts: 1
Robot.txt file is use to disallow pages which having confidential information
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
webdesigndubai is offline   Reply With Quote
Old 09-08-2016, 12:33 AM   #3
alishasmith
Registered User
 
Join Date: Aug 2016
Location: Ghaziabad
Posts: 47
Robots.txt is a text file which gives the instructions to crawlers about the website that which pages should be crawled or not.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
alishasmith is offline   Reply With Quote
Old 09-08-2016, 03:52 AM   #4
Pillars
Registered User
 
Join Date: Aug 2015
Posts: 317
Robots.txt is a text file that is inserted into your website and contains information for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Pillars is offline   Reply With Quote
Old 09-08-2016, 03:57 AM   #5
illuminatingasi
Registered User
 
Join Date: Aug 2015
Posts: 90
Robots.txt file allows to search engine spider which one page of website will be cache.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
illuminatingasi is offline   Reply With Quote
Old 09-13-2016, 12:05 AM   #6
RH-Calvin
Registered User
 
Join Date: May 2015
Location: Forum
Posts: 1,578
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
| $1 VPS Hosting

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
| Windows Server with Remote Desktop Access

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
| Servers with Free IPMI Setup
RH-Calvin is offline   Reply With Quote
Old 09-13-2016, 03:31 AM   #7
cedric903
Registered User
 
Join Date: Jul 2016
Posts: 34
Thanks, it very useful.
cedric903 is offline   Reply With Quote
Old 09-13-2016, 05:32 AM   #8
George re
Registered User
 
Join Date: Sep 2016
Posts: 33
Robots.txt is a text file which gives the instructions to crawlers about the website that which pages should be crawled or not.
George re is offline   Reply With Quote
Old 09-14-2016, 04:14 AM   #9
jasonroy21
Registered User
 
Join Date: Jul 2016
Posts: 763
Robots.txt is a text file which gives the instructions to crawlers about the website that will not allow to visit
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
jasonroy21 is offline   Reply With Quote
Old 09-19-2016, 01:59 AM   #10
AshokDixit89
Registered User
 
Join Date: Jan 2016
Location: Mumbai, India
Posts: 1,084
Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.
AshokDixit89 is offline   Reply With Quote
Old 09-19-2016, 06:11 AM   #11
lisaryan
Registered User
 
Join Date: Sep 2016
Location: Bangalore
Posts: 59
The robots.txt file helps search engines understand which pages on a website should be indexed and crawled. This file can also be used to block the unwanted crawlers to search your websites.
lisaryan is offline   Reply With Quote
Old 07-07-2017, 12:49 AM   #12
pxljobs
Registered User
 
Join Date: Jul 2016
Location: Bangalore
Posts: 376
These file give permission to search engine crawler which one to access and which one to not access these are helpful to improve the security.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
pxljobs is offline   Reply With Quote
Old 07-07-2017, 04:01 AM   #13
alexson
Registered User
 
Join Date: Jan 2017
Location: Arizona
Posts: 129
Through robots.txt you can stop search bots to crawl your websites. You can also use meta robots to prevent your website or web page.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
alexson is offline   Reply With Quote
Old 07-07-2017, 04:16 AM   #14
newproaudio
Registered User
 
Join Date: May 2017
Posts: 251
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
newproaudio is offline   Reply With Quote
Old 07-07-2017, 06:03 PM   #15
manavatmix
Registered User
 
Join Date: Jun 2016
Location: Delhi
Posts: 439
The robots file tells search engine bots which pages to index & which ones to ignore.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
manavatmix is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Robots.txt jaysh4922 Search Engine Optimization 13 07-28-2016 06:10 AM
Crawlers, spider & robots.txt? jackthomas087 Search Engine Optimization 0 04-22-2014 04:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt Fking General Discussion 1 12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress? geniusoptimizer PHP / mySQL 1 04-12-2013 01:36 PM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 03:19 AM


All times are GMT -7. The time now is 12:38 AM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.