05-06-2014, 11:02 PM | #1 |
Registered User
Join Date: Feb 2013
Location: Bangalore
Posts: 1,086
|
What is the purpose of robots.txt file?
What is the purpose of robots.txt file?
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
05-06-2014, 11:24 PM | #2 |
Registered User
Join Date: May 2014
Posts: 12
|
Robots.txt is a text file which you put on your website to tell crawler which pages you would like not to crawl.
You need to create a file and you have to enter that URL which you don't want to crawl. Have a look below. User Agent: * Disallow: //admin Disallow: ///t141187- Disallow: //file.html Disallow: //purpose
__________________
Supreme Suppliers & Manufacturers of Plastic Furniture India - Find modern designer plastic moulded good quality Premium chairs/ Furnitures,dining Chair, tables, furniture. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
05-06-2014, 11:32 PM | #3 |
Registered User
Join Date: Nov 2013
Posts: 199
|
Robots.txt is a file, which user needs to add a webpage that do not want to crawl by the spider or crawler...
eg- User Agent: * Disallow: //officeadmin Type in a notepad file and add it to website root folder ..
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
05-06-2014, 11:48 PM | #4 |
Registered User
Join Date: Feb 2014
Posts: 51
|
Robot.txt file is a text file that give instructions to search engine robots which pages you would like them not to visit.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
05-07-2014, 12:43 AM | #5 |
Registered User
Join Date: Jul 2013
Posts: 49
|
A simple text file that stops Google (and other search engines that recognize the file and its commands) from crawling the site, selected pages in the site, or selected file types in the site.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
05-07-2014, 02:38 AM | #6 |
Registered User
Join Date: Mar 2014
Posts: 210
|
Robot.txt file is a file which contains the webpages which one don't want to be crawled by the crawler.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
11-27-2017, 04:57 AM | #7 |
Registered User
Join Date: Sep 2017
Posts: 113
|
Robots.txt purpose not index for the any particular web pages in our web sites.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
11-27-2017, 05:07 AM | #8 |
Registered User
Join Date: Nov 2017
Posts: 4
|
Web site owners use the robots.txt file to give instructions about their site to web robots; this is called The Robot.txt. Exclusion Protocol tells the robot that it should not visit any pages on the site.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
11-27-2017, 08:00 AM | #9 |
Registered User
Join Date: Nov 2017
Location: Surat, Gujarat, India
Posts: 36
|
Robots.txt is a text file webmasters create to instruct web robots. how to crawl pages on their website.
|
11-27-2017, 08:16 PM | #10 |
Registered User
Join Date: Nov 2017
Posts: 342
|
Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
02-13-2018, 08:30 AM | #11 |
Registered User
Join Date: Dec 2015
Location: Bangladesh
Posts: 267
|
Robots.txt is the common name of a text file that is uploaded to a Web site's root directory and linked in the HTML code of the Web site. The robots.txt file is used to provide instructions to the Web site to Web robots and spiders. Web authors can use robots.txt
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. Outsource, To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. providing To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. services all over the world. Expert in To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. Service. |
02-13-2018, 10:25 PM | #12 |
Registered User
Join Date: Oct 2017
Posts: 306
|
The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
02-24-2018, 08:51 AM | #13 |
Registered User
Join Date: Feb 2015
Location: CA
Posts: 733
|
Robots.txt file is set of instructions help to index the website through search engines.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
02-25-2018, 09:53 PM | #14 |
Registered User
Join Date: Feb 2018
Location: Hyderabad
Posts: 47
|
robots.txt file is the file which instruct search engine which part of your webpage you don't want to get crawl by the search engine spider.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
02-25-2018, 10:56 PM | #15 |
Registered User
Join Date: Oct 2017
Posts: 306
|
Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
Thread Tools | |
Display Modes | Rate This Thread |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
The robots.txt file | rubberfender | Yahoo | 0 | 12-02-2012 07:32 PM |
RSS Feed | rajivwebads | Search Engine Optimization | 12 | 08-31-2012 07:15 AM |
Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
Don’t exceed maximum file size for robots.txt file | ClaudiaSchayffe | Search Engine Optimization | 3 | 04-02-2012 04:28 AM |