Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > Search Engine Optimization > Search Engine Optimization

Notices


Reply
 
Thread Tools Rate Thread Display Modes
Old 10-27-2016, 01:49 AM   #1
SunFineHanger
Registered User
 
Join Date: Aug 2016
Location: LITA ROAD LIPU COUNTY GUILIN GUANGXI CHINA
Posts: 30
What is the purpose of using robots.txt file in seo?

What is the purpose of using robots.txt file in seo?.....
SunFineHanger is offline   Reply With Quote

Old 10-27-2016, 02:15 AM   #2
sudeepkhana
Registered User
 
Join Date: Jan 2016
Posts: 768
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
The robots.txt file is a very powerful file if you’re working on a site’s SEO. At the same time, it also has to be used with care. It allows you to deny search engines access to certain files and folders, but that’s very often not what you want to do.
sudeepkhana is offline   Reply With Quote
Old 10-27-2016, 04:38 AM   #3
Jadanjesse
Registered User
 
Join Date: Sep 2016
Posts: 207
Before a search engine crawls your site, it will look at your robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.



To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Jadanjesse is offline   Reply With Quote
Old 11-01-2016, 12:49 AM   #4
rajkiran
Registered User
 
Join Date: Aug 2016
Posts: 459
It is important to note that malicious crawlers are likely to completely ignore robots.txt and as such, this protocol does not make a good security mechanism. Only one "Disallow:" line is allowed for each URL. Each subdomain on a root domain uses separate robots.txt files.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
rajkiran is offline   Reply With Quote
Old 11-01-2016, 01:20 AM   #5
AshokDixit89
Registered User
 
Join Date: Jan 2016
Location: Mumbai, India
Posts: 1,068
Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.
AshokDixit89 is offline   Reply With Quote
Old 11-01-2016, 07:21 AM   #6
simonsmith
Registered User
 
Join Date: Nov 2016
Posts: 9
This file is used to allow and disallow the robots of all search engines or specific search engines as well as also describing what the page robots allow to visit and index and what is not? All these details we mentioned in Robots.txt file.
simonsmith is offline   Reply With Quote
Old 11-01-2016, 08:11 AM   #7
stuartkspindlow
Registered User
 
Join Date: Sep 2014
Posts: 56
Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
stuartkspindlow is offline   Reply With Quote
Old 11-01-2016, 10:58 AM   #8
ChrisRogers123
Registered User
 
Join Date: Aug 2016
Location: USA
Posts: 119
The robots(REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
||
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
ChrisRogers123 is offline   Reply With Quote
Old 11-02-2016, 04:34 AM   #9
Harleyzoey
Registered User
 
Join Date: Jun 2016
Posts: 376
A better way to inform search engines about your will is to use a robots.txt file.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Harleyzoey is offline   Reply With Quote
Old 11-02-2016, 05:13 AM   #10
abhishek1
Registered User
 
Join Date: Oct 2016
Posts: 71
It is used to give instruction to search engine spiders how to crawl their website.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
abhishek1 is offline   Reply With Quote
Old 11-02-2016, 06:07 AM   #11
emilyoliver
Registered User
 
Join Date: Sep 2016
Posts: 249
Before a search engine crawls your site, it will look at your robots.txt file as instructions. it give instruction to search engine to crawl the site
emilyoliver is offline   Reply With Quote
Old 02-13-2018, 08:31 AM   #12
ROYCPO
Registered User
 
Join Date: Dec 2015
Location: Bangladesh
Posts: 270
Robots.txt is a text file. It is through this file, it gives instruction to search engine crawlers about indexing and caching of a webpage, file of a website or directory, domain.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Outsource
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
providing
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
services all over the world. Expert in
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Service.
ROYCPO is offline   Reply With Quote
Old 02-24-2018, 08:44 AM   #13
deepakrajput
Registered User
 
Join Date: Feb 2015
Location: CA
Posts: 742
We use robots.txt file to send indexing instructions to search engines.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
deepakrajput is offline   Reply With Quote
Old 02-25-2018, 10:02 PM   #14
RachelSofia
Registered User
 
Join Date: Dec 2017
Posts: 70
Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
RachelSofia is offline   Reply With Quote
Old 02-26-2018, 02:29 AM   #15
Kavita Agrawal
Registered User
 
Join Date: Sep 2017
Location: India
Posts: 126
Thank you very much to sharing it.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Kavita Agrawal is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
What is the purpose of robots.txt file? sahithya Search Engine Optimization 18 02-27-2018 08:54 AM
How to edit virtual robots.txt file in wordpress? geniusoptimizer PHP / mySQL 1 04-12-2013 01:36 PM
The robots.txt file rubberfender Yahoo 0 12-02-2012 07:32 PM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 03:19 AM


All times are GMT -7. The time now is 05:37 AM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.