07-26-2016, 10:24 PM
|
#1 |
|
Registered User
Join Date: Dec 2012
Posts: 657
|
Robots.txt
Robots.txt file is a special text file that is for eternity located in your Web server's root directory. Robots.txt file contains limits for Web Spiders, telling them where they have authorization to search. A Robots.txt is like significant rules for search engine spiders what to follow and what not to. Robots.txt file is a very important file if you want to have a good ranking on search engines.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
07-26-2016, 11:56 PM
|
#2 |
|
Registered User
Join Date: Jul 2016
Posts: 80
|
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
07-26-2016, 11:59 PM
|
#3 |
|
Registered User
Join Date: Mar 2016
Posts: 192
|
Robot.txt to tell google which file to index or which page to not index.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
07-27-2016, 12:04 AM
|
#4 |
|
Registered User
Join Date: Jul 2016
Posts: 27
|
Robots.txt is a name of a text file that is uploaded to a website's root directory and linked in the html code of the website. It is used to provide instructions about the website to web robots.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
07-27-2016, 12:53 AM
|
#5 |
|
Registered User
Join Date: Nov 2014
Location: India
Posts: 677
|
A robots.txt file may be a file at the foundation of your web {site} that indicates those elements of your site you don’t need accessed by computer programme crawlers. The file uses the Robots Exclusion commonplace, that may be a protocol with a tiny low set of commands which will be accustomed indicate access to your web site by section and by specific styles of internet crawlers (such as mobile crawlers vs desktop crawlers).
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
07-27-2016, 03:07 AM
|
#6 |
|
Registered User
Join Date: Jun 2016
Posts: 18
|
Robots.txt is a text file stored in a site's root directory that tells a search engine Crawler which site pages and sub-folders should not be included in the search engine Index.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
07-27-2016, 03:15 AM
|
#7 |
|
Registered User
Join Date: Jan 2016
Location: Mumbai, India
Posts: 1,064
|
Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.
Robots.txt Allowing Access to Specific Crawlers. Allow everything apart from certain patterns of URLs.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
07-27-2016, 05:41 AM
|
#8 |
|
Registered User
Join Date: Aug 2015
Location: Chennai, TN, India
Posts: 362
|
Robots.txt is a text file that is inserted into your website and contains information for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
07-27-2016, 05:42 AM
|
#9 |
|
Registered User
Join Date: Sep 2015
Location: Bangalore
Posts: 548
 |
Robots text file used to instruct the crawlers how to crawl and index pages on their website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
07-27-2016, 08:27 AM
|
#10 |
|
Registered User
Join Date: Jul 2016
Location: jeddah
Posts: 12
|
Robots.txt is a text file used by website to tell web crawler that what to follow and what to not and i found this online tool which can help you to generate robots.txt file.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. Last edited by imran.kkhan1; 07-27-2016 at 08:28 AM.. Reason: link |
|
|
|
|
07-27-2016, 11:00 PM
|
#11 |
|
Registered User
Join Date: Jun 2016
Posts: 41
|
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
07-27-2016, 11:40 PM
|
#12 |
|
Registered User
Join Date: Jul 2016
Posts: 243
|
Robots.txt is a file that should placed in your websie. It instruct to the crawler how to crawl the website and indes to the server.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
07-28-2016, 02:04 AM
|
#13 |
|
Registered User
Join Date: Jul 2016
Location: Bangalore
Posts: 370
|
The robot.txt file is an on page technique you put on your site to tell search robots which pages you would like them not to visit it give secure to your web page this process is known as robot.txt
|
|
|
|
|
07-28-2016, 05:10 AM
|
#14 |
|
Registered User
Join Date: Jul 2016
Posts: 46
|
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
| Thread Tools | |
| Display Modes | Rate This Thread |
Linear Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Crawlers, spider & robots.txt? | jackthomas087 | Search Engine Optimization | 0 | 04-22-2014 03:29 AM |
| Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
| How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
| The robots.txt file | rubberfender | Yahoo | 0 | 12-02-2012 07:32 PM |
| Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
All times are GMT -7. The time now is 02:37 AM.