07-15-2016, 05:44 AM | #1 |
Registered User
Join Date: Feb 2016
Posts: 72
|
What is robots.txt used for?
What is robots.txt used for?
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , |
07-15-2016, 09:15 AM | #2 |
Registered User
Join Date: Jun 2016
Posts: 3
|
robots.txt file is used for control search engine bot.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-15-2016, 11:32 PM | #3 |
Registered User
Join Date: Aug 2015
Location: Chennai, TN, India
Posts: 362
|
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-16-2016, 03:41 AM | #4 |
Registered User
Join Date: Jul 2016
Posts: 80
|
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-16-2016, 03:59 AM | #5 |
Registered User
Join Date: Apr 2016
Location: pune
Posts: 583
|
Robots.txt file controls how search engine spiders see and interact with your web pages.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-19-2016, 05:51 AM | #6 |
Registered User
Join Date: Jul 2016
Posts: 89
|
Robots.txt is a file, that instruct to the crawler what pages are to be crawl and index it in to the server
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-19-2016, 06:02 AM | #7 |
Registered User
Join Date: Jul 2016
Location: Singapore
Posts: 31
|
The robots.txt file is allow to give instructions about site to web robots. One can control his page indexing by this robots.txt. It's very important.
|
07-20-2016, 12:48 AM | #8 |
Registered User
Join Date: Jun 2016
Posts: 135
|
Robot.txt is a text file that is uploaded to a website's root directory and linked in the html code of the website. A simple file that contains components used to specify the pages on a website that must be crawled by search engine bots.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-20-2016, 01:49 AM | #9 |
Registered User
Join Date: Aug 2015
Location: Chennai, TN, India
Posts: 362
|
Robots.txt is a text file that is inserted into your website and contains information for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-20-2016, 01:49 AM | #10 |
Registered User
Join Date: Jul 2016
Posts: 34
|
Website communication with web crawler and other web robots. this is basic information for Robot.txt
__________________
Tel Aviv Apartments | Tel Aviv Apartment Rentals | Holiday Rentals Tel Aviv |
07-20-2016, 01:57 AM | #11 |
Registered User
Join Date: Jul 2016
Location: Bangalore
Posts: 210
|
The robot.txt is one of the on page optimization technique it give the permission to search engine crawler whether allow or disallow spider index those pages so uses of robot.txt file it give protection and give security to your secret pages
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
11-16-2016, 11:02 PM | #12 |
Registered User
Join Date: Oct 2016
Posts: 580
|
Robot.txt is a text file that is uploaded to a website's root directory and linked in the html code of the website. A simple file that contains components used to specify the pages on a website that must be crawled by search engine bots.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
11-17-2016, 01:42 AM | #13 |
Registered User
Join Date: Jan 2016
Location: India
Posts: 1,258
|
Robots.txt is a text (not HTML) file you put on your site to tell search robots which pages you would like them not to visit. The robots.txt is a very simple text file that is placed in your root directory. It creates to instruct the robots (typically search engine robots) how to crawl and index pages on their website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
11-17-2016, 01:47 AM | #14 |
Registered User
Join Date: Oct 2016
Location: 12102 5th St Houston, TX 77072, USA
Posts: 2
|
robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
11-17-2016, 02:05 AM | #15 |
Registered User
Join Date: Jul 2016
Location: USA
Posts: 138
|
robots.txt is a simple text file webmaster create to instruct search engine robots how to crawl and indexed of their website
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Crawlers, spider & robots.txt? | jackthomas087 | Search Engine Optimization | 0 | 04-22-2014 03:29 AM |
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
The robots.txt file | rubberfender | Yahoo | 0 | 12-02-2012 07:32 PM |
Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |