06-02-2016, 01:05 AM | #1 |
Registered User
Join Date: Jan 2016
Location: Mumbai, India
Posts: 1,064
|
What is the use of Robots.txt..?
What is the use of Robots.txt..?
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
06-02-2016, 02:08 AM | #2 |
Registered User
Join Date: May 2016
Posts: 36
|
"Robots.txt" redirects here. For Wikipedia's robots.txt files, see the MediaWiki Robots.txt file and the English Wikipedia Robots.txt file.
The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned. Robots are often used by search engines to categorize web sites. Not all robots cooperate with the standard; email harvesters, spambots, malware and Search Engines robots that scan for security vulnerabilities may even start with the portions of the website where they have been told to stay out. The standard is different from, but can be used in conjunction with Sitemaps, a robot inclusion standard for websites.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - – It is a launch platform which allows you to create a JV page |
06-02-2016, 04:32 AM | #3 |
Registered User
Join Date: May 2016
Location: Florida, United State
Posts: 36
|
Robot.txt is a text file. It gives instruction to search engine crawlers about indexing and caching of a web page, file of a website or directory, domain.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
06-02-2016, 04:44 AM | #4 |
Registered User
Join Date: May 2016
Posts: 551
|
When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
06-02-2016, 05:33 AM | #5 |
Registered User
Join Date: Jun 2016
Location: lebanon
Posts: 8
|
Website owners use the /robots.txt file to give instructions about their website to robots. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
|
06-02-2016, 06:01 AM | #6 |
Registered User
Join Date: May 2016
Posts: 14
|
it will guide google spiders to crawl your site properly. it will tell to the spiders not to verify specific folder or file in your blog.
__________________
pawan kumar SEO guide To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
06-02-2016, 06:09 AM | #7 |
Registered User
Join Date: May 2016
Posts: 60
|
Utilizing a robots.txt document is simple, yet requires access to your server's root area. Case in point, if your site is situated at:
http://adomain.com/mysite/index.html you should have the capacity to make a record situated here: http://adomain.com/robots.txt
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
06-02-2016, 10:40 PM | #8 |
Registered User
Join Date: Jun 2016
Posts: 135
|
Robots.txt is a text file that is uploaded to a website's root directory and linked in the html code of the website. The robots.txt is used to provide instructions about the website to web robots and spiders.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
06-03-2016, 02:05 AM | #9 |
Registered User
Join Date: Aug 2015
Location: Chennai, TN, India
Posts: 362
|
Robots.txt is a text file that is inserted into your website and contains information for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Crawlers, spider & robots.txt? | jackthomas087 | Search Engine Optimization | 0 | 04-22-2014 03:29 AM |
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
The robots.txt file | rubberfender | Yahoo | 0 | 12-02-2012 07:32 PM |
Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |