Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > Internet Promotion > Web Promotion

Notices


Reply
 
Thread Tools Rating: Thread Rating: 7 votes, 2.57 average. Display Modes
Old 12-13-2017, 09:06 PM   #16
aliceta
Registered User
 
Join Date: Oct 2017
Posts: 92
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
__________________
How to
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

How to
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
aliceta is offline   Reply With Quote
Old 12-14-2017, 06:20 AM   #17
sirishasiri
Registered User
 
Join Date: Dec 2017
Posts: 58
Robot.txt

Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.
Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display.
sirishasiri is offline   Reply With Quote
Old 01-03-2018, 04:37 AM   #18
Alex Hale
Registered User
 
Join Date: Jan 2018
Posts: 10
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. ... The "Disallow: /" tells the robot that it should not visit any pages on the site.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Alex Hale is offline   Reply With Quote
Old 01-03-2018, 04:46 AM   #19
stellasweety
Registered User
 
Join Date: Mar 2017
Posts: 666
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
stellasweety is offline   Reply With Quote
Old 03-13-2018, 03:38 AM   #20
successmantra
Registered User
 
Join Date: Aug 2017
Location: batala
Posts: 128
Robots.txt file allows to search engine spider which one page of website will be cache.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
successmantra is offline   Reply With Quote
Old 03-16-2018, 04:11 AM   #21
Yashmin
Registered User
 
Join Date: Mar 2017
Location: Dubai
Posts: 151
Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.


To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Yashmin is offline   Reply With Quote
Old 03-21-2018, 04:23 AM   #22
thesslstoreindi
Registered User
 
Join Date: Mar 2018
Posts: 12
Web site owners use the robots.txt file to give instructions about their site to web robots
thesslstoreindi is offline   Reply With Quote
Old 03-29-2018, 07:44 AM   #23
techguru
Registered User
 
Join Date: Mar 2018
Location: london
Posts: 6
robots.txt is a text file which helps to navigate, search bots to allow or disallow in your files. This file use for giving instruction to crawlers in website to disallow in your confidential files
techguru is offline   Reply With Quote
Old 04-02-2018, 03:28 AM   #24
multai
Registered User
 
Join Date: Apr 2018
Posts: 63
Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines.

http://www.multaisolutions.com/web.html
multai is offline   Reply With Quote
Old 04-02-2018, 03:53 AM   #25
stellasweety
Registered User
 
Join Date: Mar 2017
Posts: 666
Robots.txt is a content record which gives the guidelines to crawlers about the site what pages ought to be crept or not.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
stellasweety is offline   Reply With Quote
Old 06-26-2018, 05:48 AM   #26
mrseoranker
Registered User
 
Join Date: Jun 2018
Posts: 9
Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display.
SEO Training Lahore
SEO Services in Lahore
PPC Expert in Lahore
mrseoranker is offline   Reply With Quote
Old 06-27-2018, 03:18 AM   #27
EmilyPete
Registered User
 
Join Date: May 2016
Posts: 92
Robots.txt File tells search engine which pages to Index and which pages not to Index. Webmasters have full control of website pages which for crawling the website.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
EmilyPete is offline   Reply With Quote
Old 06-28-2018, 04:53 AM   #28
omerahmed
Registered User
 
Join Date: Jun 2018
Location: Pakistan
Posts: 5
Robots.txt is a text file that tells the crawler not to visit certain pages or posts in the website. It is used to mention the part of website that web master does not want the search engines to crawl.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
omerahmed is offline   Reply With Quote
Old 07-14-2018, 06:39 AM   #29
garimasharmapla
Registered User
 
Join Date: Jun 2018
Location: New Delhi
Posts: 13
robots.txt is a file that you upload on your root domain, it helps crawler to index your most effectively. By this file you tell crawler to nofollow the page or directory on your site .
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
garimasharmapla is offline   Reply With Quote
Old 07-14-2018, 11:02 PM   #30
assignmenthelp7
Registered User
 
Join Date: Jul 2018
Posts: 5
We use robots.txt file to give instructions about their site to web crawler, this is called The Robots Exclusion Protocol. The User-agent: * means this section applies to all robots. The Disallow: / tells the robot that it should not visit any pages on the site.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
assignmenthelp7 is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Robots.txt jaysh4922 Search Engine Optimization 13 07-28-2016 06:10 AM
Crawlers, spider & robots.txt? jackthomas087 Search Engine Optimization 0 04-22-2014 04:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt Fking General Discussion 1 12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress? geniusoptimizer PHP / mySQL 1 04-12-2013 01:36 PM
Unable to delete robots.txt file notfake Yahoo 0 04-29-2012 03:19 AM


All times are GMT -7. The time now is 12:36 PM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.