Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > Internet Promotion > Web Promotion

Notices


Reply
 
Thread Tools Rate Thread Display Modes
Old 07-07-2017, 06:03 PM   #16
manavatmix
Registered User
 
Join Date: Jun 2016
Location: Delhi
Posts: 510
The robots file tells search engine bots which pages to index & which ones to ignore.
manavatmix is offline   Reply With Quote
Old 12-13-2017, 09:06 PM   #17
aliceta
Registered User
 
Join Date: Oct 2017
Posts: 116
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
aliceta is offline   Reply With Quote
Old 12-14-2017, 06:20 AM   #18
sirishasiri
Registered User
 
Join Date: Dec 2017
Posts: 123
Robot.txt

Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.
Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display.
sirishasiri is offline   Reply With Quote
Old 01-03-2018, 04:37 AM   #19
Alex Hale
Registered User
 
Join Date: Jan 2018
Posts: 10
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. ... The "Disallow: /" tells the robot that it should not visit any pages on the site.
Alex Hale is offline   Reply With Quote
Old 01-03-2018, 04:46 AM   #20
stellasweety
Registered User
 
Join Date: Mar 2017
Posts: 647
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
stellasweety is offline   Reply With Quote
Old 01-05-2018, 02:50 AM   #21
albertomorrison
Registered User
 
Join Date: Dec 2017
Location: Santa Clara, California, USA
Posts: 23
How does it work?
Before a search engine crawls your site, it will look at your robots.txt file as instructions on where they are allowed to crawl (visit) and index (save) on the search engine results.
Robots.txt files are useful:
If you want search engines to ignore any duplicate pages on your website
If you don’t want search engines to index your internal search results pages
If you don’t want search engines to index certain areas of your website or a whole website
If you don’t want search engines to index certain files on your website (images, PDFs, etc.)
If you want to tell search engines where your sitemap is located
albertomorrison is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Robots.txt jaysh4922 Search Engine Optimization 15 07-29-2016 06:07 AM
Crawlers, spider & robots.txt? jackthomas087 Search Engine Optimization 0 04-22-2014 04:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt Fking General Discussion 1 12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress? geniusoptimizer PHP / mySQL 1 04-12-2013 01:36 PM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 03:19 AM


All times are GMT -7. The time now is 05:10 AM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.