09-30-2016, 05:19 AM
|
#1 |
|
Registered User
Join Date: Jul 2016
Posts: 43
|
What is robots.txt?
Robots.txt is a text file. It is through this file, it gives instruction to search engine crawlers about indexing and caching of a webpage, file of a website or directory, domain.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
09-30-2016, 04:08 PM
|
#2 |
|
Registered User
Join Date: Aug 2016
Posts: 109
|
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too naïve to rely on robots.txt to protect it from being indexed and displayed in search results.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
09-30-2016, 11:34 PM
|
#3 |
|
Registered User
Join Date: Sep 2013
Posts: 411
|
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
10-01-2016, 12:31 AM
|
#5 | |
|
Registered User
Join Date: Jun 2016
Posts: 99
|
Quote:
.................................................. .................................................. ............................. (bai viet rat huu ich !) .................................................. .................................................. .............................. dây điện cadivi dây cáp điện cadivi
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
|
10-01-2016, 01:29 AM
|
#6 |
|
Registered User
Join Date: Apr 2016
Location: pune
Posts: 583
|
Robot.txt allow search bots to read or not read the website files, folder or content.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
10-01-2016, 03:24 AM
|
#7 |
|
Registered User
Join Date: Sep 2016
Location: Bangalore
Posts: 21
|
Robots.txt enables you to instruct search engine machine to crawl and index pages on your website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
10-01-2016, 09:32 AM
|
#8 |
|
Registered User
Join Date: Aug 2016
Location: USA
Posts: 117
|
It is a type of text file which tells bot what to crawl or what to not ?
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
10-01-2016, 01:38 PM
|
#9 |
|
Registered User
Join Date: Sep 2016
Posts: 9
|
The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
__________________
ADMIN Middo To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
10-03-2016, 08:29 PM
|
#10 |
|
Registered User
Join Date: Aug 2016
Posts: 89
|
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - What is it? To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - what's legit or scam? |
|
|
|
|
10-03-2016, 10:39 PM
|
#11 |
|
Registered User
Join Date: Dec 2015
Posts: 277
|
Well robots.txt is actually a notepad file in which we create instructions/commands for web crawler to not check out and index the link. It is used when the web page is new and there is no content in the web page, therefore it is important for SEO.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
10-04-2016, 12:21 AM
|
#12 |
|
Registered User
Join Date: Apr 2013
Posts: 382
|
A robots.txt file is a file at the root of your site that indicates those parts of your site you don't want accessed by search engine crawlers.
|
|
|
|
|
10-04-2016, 02:46 AM
|
#13 |
|
Registered User
Join Date: Feb 2016
Posts: 35
|
Robot.txt is a text file which is used to not to index a particular page. It is put in root file.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
10-04-2016, 03:07 AM
|
#14 |
|
Registered User
Join Date: Jul 2016
Posts: 241
|
* A robots.txt file is a file at the root of your site that indicates those parts of your site you don’t want to be accessed by search engine crawlers. The file uses the Robots Exclusion Standard, which is a protocol with a small set of commands that can be used to indicate access to your site by section and by specific kinds of web crawlers (such as mobile crawlers vs desktop crawlers).
* Robots.txt is the common name of a text file that is uploaded to a Web site's root directory and linked in the HTML code of the Web site. The robots.txt file is used to provide instructions to the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
10-04-2016, 04:26 AM
|
#15 |
|
Registered User
Join Date: Oct 2016
Location: Noida
Posts: 10
|
A robots.txt file is used to notify search engine to index or not to index webpages of the website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Linear Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Robots.txt | jaysh4922 | Search Engine Optimization | 13 | 07-28-2016 05:10 AM |
| Crawlers, spider & robots.txt? | jackthomas087 | Search Engine Optimization | 0 | 04-22-2014 03:29 AM |
| Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
| How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
| Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
All times are GMT -7. The time now is 04:03 AM.