Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > Search Engine Optimization > Search Engine Optimization

Notices


Reply
 
Thread Tools Rate Thread Display Modes
Old 10-27-2016, 12:49 AM   #1
SunFineHanger
Registered User
 
Join Date: Aug 2016
Location: LITA ROAD LIPU COUNTY GUILIN GUANGXI CHINA
Posts: 32
What is the purpose of using robots.txt file in seo?

What is the purpose of using robots.txt file in seo?.....
SunFineHanger is offline   Reply With Quote

Old 10-27-2016, 01:15 AM   #2
sudeepkhana
Registered User
 
Join Date: Jan 2016
Posts: 881
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
The robots.txt file is a very powerful file if you’re working on a site’s SEO. At the same time, it also has to be used with care. It allows you to deny search engines access to certain files and folders, but that’s very often not what you want to do.
__________________
seo company in bhilai
sudeepkhana is offline   Reply With Quote
Old 10-27-2016, 03:38 AM   #3
Jadanjesse
Registered User
 
Join Date: Sep 2016
Posts: 234
Before a search engine crawls your site, it will look at your robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results.
Jadanjesse is offline   Reply With Quote
Old 10-31-2016, 11:49 PM   #4
rajkiran
Registered User
 
Join Date: Aug 2016
Posts: 499
It is important to note that malicious crawlers are likely to completely ignore robots.txt and as such, this protocol does not make a good security mechanism. Only one "Disallow:" line is allowed for each URL. Each subdomain on a root domain uses separate robots.txt files.
__________________
car cover || car covers
rajkiran is offline   Reply With Quote
Old 11-01-2016, 12:20 AM   #5
AshokDixit89
Registered User
 
Join Date: Jan 2016
Location: Mumbai, India
Posts: 1,161
Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.
AshokDixit89 is offline   Reply With Quote
Old 11-01-2016, 05:18 AM   #6
sadianisar
Registered User
 
Join Date: Jul 2016
Posts: 3,465
It is used for newly built websites and in SEO, it is used to stop crawler to not visit or follow the link/webpage which we asked crawler in the form of instructions in robots.txt file.
sadianisar is offline   Reply With Quote
Old 11-01-2016, 06:21 AM   #7
simonsmith
Registered User
 
Join Date: Nov 2016
Posts: 10
This file is used to allow and disallow the robots of all search engines or specific search engines as well as also describing what the page robots allow to visit and index and what is not? All these details we mentioned in Robots.txt file.
simonsmith is offline   Reply With Quote
Old 11-01-2016, 07:11 AM   #8
stuartkspindlow
Registered User
 
Join Date: Sep 2014
Posts: 62
Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.
__________________
Payroll Services In London
stuartkspindlow is offline   Reply With Quote
Old 11-01-2016, 09:58 AM   #9
ChrisRogers123
Registered User
 
Join Date: Aug 2016
Location: USA
Posts: 133
The robots(REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
ChrisRogers123 is offline   Reply With Quote
Old 11-02-2016, 03:34 AM   #10
Harleyzoey
Registered User
 
Join Date: Jun 2016
Posts: 416
A better way to inform search engines about your will is to use a robots.txt file.
Harleyzoey is offline   Reply With Quote
Old 11-02-2016, 04:13 AM   #11
abhishek1
Registered User
 
Join Date: Oct 2016
Posts: 83
It is used to give instruction to search engine spiders how to crawl their website.
abhishek1 is offline   Reply With Quote
Old 11-02-2016, 05:07 AM   #12
emilyoliver
Registered User
 
Join Date: Sep 2016
Posts: 358
Before a search engine crawls your site, it will look at your robots.txt file as instructions. it give instruction to search engine to crawl the site
emilyoliver is offline   Reply With Quote
Old 02-13-2018, 07:31 AM   #13
ROYCPO
Registered User
 
Join Date: Dec 2015
Location: Bangladesh
Posts: 359
Robots.txt is a text file. It is through this file, it gives instruction to search engine crawlers about indexing and caching of a webpage, file of a website or directory, domain.
__________________
Clipping Path Outsource, digital graphics studio providing image editing services all over the world. Expert in clipping path Service.
ROYCPO is offline   Reply With Quote
Old 02-24-2018, 07:44 AM   #14
deepakrajput
Registered User
 
Join Date: Feb 2015
Location: CA
Posts: 991
We use robots.txt file to send indexing instructions to search engines.
deepakrajput is offline   Reply With Quote
Old 02-25-2018, 09:02 PM   #15
RachelSofia
Registered User
 
Join Date: Dec 2017
Posts: 117
Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.
RachelSofia is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
What is the purpose of robots.txt file? sahithya Search Engine Optimization 18 02-27-2018 07:54 AM
Apple: a new file system encrypted by nature bolhaw General Discussion 0 06-17-2016 07:25 AM
How to edit virtual robots.txt file in wordpress? geniusoptimizer PHP / mySQL 1 04-12-2013 12:36 PM
The robots.txt file rubberfender Yahoo 0 12-02-2012 06:32 PM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 02:19 AM


All times are GMT -7. The time now is 10:03 PM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.