03-24-2019, 09:37 PM
|
#1 |
|
Registered User
Join Date: Sep 2017
Posts: 224
|
What is robots.txt?
What is robots.txt?
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
03-25-2019, 12:04 AM
|
#2 |
|
Registered User
Join Date: Feb 2018
Posts: 186
|
The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. ... The slash after “Disallow” tells the robot to not visit any pages on the site.
More Details Contact Hourglass IT Solutions Our Services: Web Design Company | Digital Marketing Services Chennai | Seo Company in Chennai | Mobile App Development Company Chennai | Ecommerce Web Design Services
__________________
HourGlass IT Solutions |
|
|
|
|
03-25-2019, 12:34 AM
|
#3 |
|
Registered User
Join Date: Aug 2016
Posts: 961
|
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
|
|
|
|
|
03-25-2019, 03:44 AM
|
#4 |
|
Registered User
Join Date: Nov 2015
Location: USA
Posts: 531
|
Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl & index pages on their website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-25-2019, 03:55 AM
|
#5 |
|
Registered User
Join Date: Feb 2018
Location: Bangalore
Posts: 283
|
Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
Basic format of the robot.txt file. User-agent: [user-agent name] Disallow: [URL string not to be crawled]
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-25-2019, 05:20 AM
|
#6 |
|
Registered User
Join Date: Nov 2012
Location: India
Posts: 276
 |
Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Shopping Cart- PrestaShop, Zencart; CMS - WordPress, Joomla; Custom Development To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - SEO, Pay Per Click (PPC), SEM, Link Popularity Building, SEO Copywriting, SMO, SMM |
|
|
|
|
03-27-2019, 01:25 PM
|
#7 |
|
Registered User
Join Date: Mar 2018
Location: Chennai
Posts: 78
|
It is a webmaster tool.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-27-2019, 11:05 PM
|
#8 |
|
Registered User
Join Date: Apr 2016
Posts: 83
|
What is robots.txt?
The robots.txt file is a text file, placed at the root of a website. It makes it possible to indicate to search engine robots areas not to explore: web pages or complete directories.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-27-2019, 11:30 PM
|
#9 |
|
Registered User
Join Date: Jan 2016
Location: Hyderabad
Posts: 4,287
|
Web site owners use the robots file to give instructions about their site to web robots.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
|
|
|
|
03-27-2019, 11:39 PM
|
#10 |
|
Registered User
Join Date: May 2015
Location: Forum
Posts: 2,517
|
Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.
__________________
█ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | $1 VPS Hosting █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | Windows Server with Remote Desktop Access █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | Servers with Free IPMI Setup |
|
|
|
|
04-02-2019, 03:04 AM
|
#11 |
|
Registered User
Join Date: Oct 2017
Location: India
Posts: 39
|
Robots.txt file is at the root of the website that involves sectors of your website you don’t want to be attained by search engine crawlers. Webmasters use a robot.txt file to instruct the search engine robots on how to crawl & index the web pages.
|
|
|
|
 |
«
Previous Thread
|
Next Thread
»
| Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
| Thread Tools | |
| Display Modes | Rate This Thread |
Linear Mode
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Robots.txt or Meta Robots.txt??? | santoshkumar | Search Engine Optimization | 10 | 02-14-2017 04:03 AM |
| Robots.txt | jaysh4922 | Search Engine Optimization | 13 | 07-28-2016 05:10 AM |
| Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
| How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
| Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |
All times are GMT -7. The time now is 01:56 AM.