05-02-2017, 04:57 AM
|
#1 |
|
Registered User
Join Date: Jun 2014
Posts: 138
|
What is a robots.txt file?
hello friends tell me
What is a robots.txt file?
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
05-02-2017, 10:17 PM
|
#2 |
|
Registered User
Join Date: Dec 2016
Posts: 377
|
Robots.txt is a text file. It is through this file, it gives instruction to search engine crawlers about indexing and caching of a webpage, file of a website or directory, domain...
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
05-02-2017, 10:21 PM
|
#3 |
|
Registered User
Join Date: Sep 2016
Posts: 37
|
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
05-02-2017, 10:51 PM
|
#4 |
|
Registered User
Join Date: Apr 2017
Posts: 87
|
Robot.txt is a file which is actually a text file created to instruct robots (search engine robots) that how and which pages to crawl of a website.
|
|
|
|
|
09-01-2017, 05:22 AM
|
#5 |
|
Registered User
Join Date: Jul 2017
Posts: 229
|
In a nutshell. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. ... The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
09-01-2017, 10:12 PM
|
#6 |
|
Registered User
Join Date: Jul 2017
Posts: 36
|
This file is only visible to googlebots as it commands google bot to crawl allowed pages which admin has allowed
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
09-05-2017, 12:37 AM
|
#7 |
|
Registered User
Join Date: Jun 2017
Posts: 184
|
In a nutshell. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. ... The "Disallow: /" tells the robot that it should not visit any pages on the site.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
09-05-2017, 01:35 AM
|
#8 |
|
Registered User
Join Date: Nov 2016
Posts: 515
|
Robots.txt is a content (not html) record you put on your site to tell look robots which pages you might want them not to visit.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Linear Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
| The robots.txt file | rubberfender | Yahoo | 0 | 12-02-2012 07:32 PM |
| Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
| Don’t exceed maximum file size for robots.txt file | ClaudiaSchayffe | Search Engine Optimization | 3 | 04-02-2012 04:28 AM |
All times are GMT -7. The time now is 09:41 AM.