Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > Search Engine Optimization > Search Engine Optimization

Notices


Reply
 
Thread Tools Rate Thread Display Modes
Old 11-28-2017, 11:20 PM   #31
scrapcarsuk
Registered User
 
Join Date: Jun 2017
Posts: 485
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
scrapcarsuk is offline   Reply With Quote
Old 11-28-2017, 11:20 PM   #32
scrapcarsuk1234
Registered User
 
Join Date: Jun 2017
Posts: 420
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
scrapcarsuk1234 is offline   Reply With Quote
Old 12-09-2017, 10:34 PM   #33
cuong25081995
Registered User
 
Join Date: Nov 2017
Posts: 500
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
cuong25081995 is offline   Reply With Quote
Old 12-10-2017, 10:15 PM   #34
leenahahim
Registered User
 
Join Date: Feb 2016
Posts: 227
robots.txt is text file it helps to block all web crawler from all pages or specified web crawler from specific web page.
leenahahim is offline   Reply With Quote
Old 12-10-2017, 10:16 PM   #35
rajapriya
Registered User
 
Join Date: Nov 2017
Posts: 54
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
rajapriya is offline   Reply With Quote
Old 02-12-2018, 12:59 AM   #36
Jessianut
Registered User
 
Join Date: Jul 2017
Posts: 465
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. ... The "Disallow: /" tells the robot that it should not visit any pages on the site.
Jessianut is offline   Reply With Quote
Old 02-13-2018, 07:29 AM   #37
ROYCPO
Registered User
 
Join Date: Dec 2015
Location: Bangladesh
Posts: 359
Robots.txt is a text file that is inserted into your website to contain instructions for search engine spiders. The file lists web pages that are allowed and disallowed from search engine crawling.
__________________
Clipping Path Outsource, digital graphics studio providing image editing services all over the world. Expert in clipping path Service.
ROYCPO is offline   Reply With Quote
Old 02-14-2018, 01:10 AM   #38
k2bseo
Registered User
 
Join Date: Dec 2017
Posts: 202
Whenever we need certain pages not to be crawled by search engine, we can block those pages in robots file
k2bseo is offline   Reply With Quote
Old 02-14-2018, 03:04 AM   #39
triarch
Registered User
 
Join Date: Feb 2018
Posts: 21
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.
triarch is offline   Reply With Quote
Old 02-14-2018, 03:09 AM   #40
Elizabethezhil
Registered User
 
Join Date: Aug 2017
Posts: 451
Website owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
Elizabethezhil is offline   Reply With Quote
Old 02-14-2018, 03:54 AM   #41
nathanleo
Registered User
 
Join Date: Jul 2017
Posts: 378
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.
nathanleo is offline   Reply With Quote
Old 02-14-2018, 04:11 AM   #42
Navjot Singh
Registered User
 
Join Date: Aug 2017
Posts: 170
The purpose of a robots.txt file, also known as the robots exclusion protocol, is to give webmasters control over what pages robots can crawl and index on their site.
__________________
We are one of the leading Health Management Company in Delhi NCR with NABL accredited & ISO certified lab.
Navjot Singh is offline   Reply With Quote
Old 02-14-2018, 04:17 AM   #43
irsgroup
Registered User
 
Join Date: Dec 2017
Posts: 25
Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website.
irsgroup is offline   Reply With Quote
Old 02-14-2018, 04:21 AM   #44
Adya
Registered User
 
Join Date: Jul 2017
Posts: 545
Robot.txt is the file included in the site which indicate Crawler about which pages of the site will be crawled and which not.
Adya is offline   Reply With Quote
Old 02-14-2018, 11:28 PM   #45
lilianalilly
Registered User
 
Join Date: Apr 2017
Posts: 578
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.
lilianalilly is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
What is the purpose of robots.txt file? sahithya Search Engine Optimization 18 02-27-2018 07:54 AM
What is the purpose of using robots.txt file in seo? SunFineHanger Search Engine Optimization 15 02-26-2018 01:29 AM
Apple: a new file system encrypted by nature bolhaw General Discussion 0 06-17-2016 07:25 AM
How to edit virtual robots.txt file in wordpress? geniusoptimizer PHP / mySQL 1 04-12-2013 12:36 PM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 02:19 AM


All times are GMT -7. The time now is 02:07 AM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.