12-06-2017, 06:18 AM
|
#16 |
|
Registered User
Join Date: Dec 2017
Posts: 57
|
Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.
|
|
|
|
12-06-2017, 06:19 AM
|
#17 |
|
Registered User
Join Date: Jun 2017
Posts: 23
|
robots.txt is mainly used to allow google crawlers for indexing purpose
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
12-06-2017, 06:45 AM
|
#18 |
|
Registered User
Join Date: Dec 2015
Location: Bangladesh
Posts: 267
 |
Website owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. Outsource, To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. providing To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. services all over the world. Expert in To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. Service. |
|
|
|
|
12-06-2017, 11:14 PM
|
#19 |
|
Registered User
Join Date: Nov 2016
Location: India
Posts: 78
|
Robots txt in generally for use information our crawlers website.
|
|
|
|
|
12-08-2017, 09:48 PM
|
#20 |
|
Registered User
Join Date: Feb 2016
Posts: 454
|
I helps to block all web crawler form all webpages or specific web crawler form specif webpage.
|
|
|
|
|
12-09-2017, 02:04 AM
|
#21 |
|
Registered User
Join Date: Oct 2015
Location: Agra
Posts: 232
|
Robot txt is the exclusion which is used the proper scanned files.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
12-09-2017, 01:07 PM
|
#22 |
|
Registered User
Join Date: Dec 2017
Location: hydarabad
Posts: 37
|
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
11-26-2021, 10:23 PM
|
#23 |
|
Registered User
Join Date: Jul 2017
Posts: 2,060
|
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Linear Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Robots.txt or Meta Robots.txt??? | santoshkumar | Search Engine Optimization | 10 | 02-14-2017 04:03 AM |
| Robots.txt | jaysh4922 | Search Engine Optimization | 13 | 07-28-2016 05:10 AM |
| Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
| How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
| Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
All times are GMT -7. The time now is 11:46 AM.