03-09-2016, 12:31 AM | #1 |
Registered User
Join Date: Sep 2014
Location: california
Posts: 52
|
What is robots.txt used for?
I learnt here about sitemap.xml file. Now I'm little bit confused for what is robots.txt used for?
Please guys clear it.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-10-2016, 01:07 AM | #2 |
Registered User
Join Date: May 2015
Location: Forum
Posts: 2,517
|
Robots.txt is a text file that is used to insert into the website. This file contains URLs that are allowed and disallowed from search engine crawling.
__________________
█ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | $1 VPS Hosting █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | Windows Server with Remote Desktop Access █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | Servers with Free IPMI Setup |
03-12-2016, 04:48 AM | #3 |
Registered User
Join Date: Feb 2016
Posts: 20
|
The purpose of the using robots.txt file is to tell the search engine bots which file should and which should not be indexed by them. In order to allow search bots to crawl and index the entire content of your website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-14-2016, 12:01 AM | #4 |
Registered User
Join Date: Jan 2016
Location: India
Posts: 1,258
|
Robot.txt file Is used to tell the search engine which Page you do not Want Crawled or Indexed.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-14-2016, 05:13 AM | #5 |
Registered User
Join Date: Jan 2016
Location: Thailand
Posts: 32
|
Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders.
|
03-14-2016, 12:53 PM | #6 |
Registered User
Join Date: Mar 2016
Posts: 256
|
Google search bro
|
03-14-2016, 11:25 PM | #7 |
Registered User
Join Date: Jun 2014
Location: India
Posts: 127
|
Its very easy to understand here we give instructions to search engine which page they can crawl and index. And get more detail from online blogs.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Crawlers, spider & robots.txt? | jackthomas087 | Search Engine Optimization | 0 | 04-22-2014 03:29 AM |
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
The robots.txt file | rubberfender | Yahoo | 0 | 12-02-2012 07:32 PM |
Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |