|
Notices |
|
Thread Tools | Rate Thread | Display Modes |
11-28-2016, 11:26 PM | #16 |
Registered User
Join Date: Sep 2016
Posts: 75
|
Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
11-28-2016, 11:40 PM | #17 |
Registered User
Join Date: Sep 2016
Posts: 70
|
The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
11-28-2016, 11:45 PM | #18 |
Registered User
Join Date: Oct 2016
Posts: 81
|
Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
11-28-2016, 11:51 PM | #19 |
Registered User
Join Date: Jul 2016
Posts: 243
|
Robots.txt file is for giving instruction to the crawlers indicates which one to crawl and which are don't crawl.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
11-29-2016, 04:10 AM | #20 |
Registered User
Join Date: Aug 2016
Posts: 187
|
Robot.txt used to provide guidance to the search engine crawler about the web pages that which page to be crawled and which is not.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-14-2016, 12:43 AM | #21 |
Registered User
Join Date: Dec 2016
Posts: 94
|
The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way. So, if you don't put robots.txt in the right place, do not be surprised that search engines index your whole site.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-14-2016, 02:44 AM | #22 |
Registered User
Join Date: Aug 2016
Location: USA
Posts: 117
|
It is a type of text file which allows search engine to what to crawl or what to not
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-14-2016, 03:13 AM | #23 |
Registered User
Join Date: Aug 2016
Location: London
Posts: 115
|
Robots.txt is document you put on your site to tell look robots which pages you might want them Do and Don't to visit.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-14-2016, 04:14 AM | #24 |
Registered User
Join Date: Feb 2016
Location: Green Cove Springs, FL
Posts: 321
|
Robots.txt file is a file through which we give instructions to the search engines whether to crawl or not the particular web page.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-15-2016, 12:54 AM | #25 |
Registered User
Join Date: Oct 2016
Posts: 81
|
Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which are not to be crawled.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-15-2016, 03:35 AM | #26 |
Registered User
Join Date: Dec 2016
Location: New York
Posts: 113
|
Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not.
__________________
Claim ME now anytime to get
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-16-2016, 12:23 AM | #27 |
Registered User
Join Date: Aug 2016
Posts: 205
|
robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-17-2016, 01:49 AM | #28 |
Registered User
Join Date: Sep 2016
Posts: 617
|
The purpose of a robots.txt file is to tell search engines not to crawl the contents of specific directories or even individual pages within a website. ... However there are a couple of reasons why all websites would benefit from having one. The first thing search engines look for is the robots.txt file.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-17-2016, 05:22 AM | #29 |
Registered User
Join Date: Sep 2016
Posts: 70
|
Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-19-2016, 01:44 AM | #30 |
Registered User
Join Date: Sep 2016
Posts: 171
|
It is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
|
|