07-02-2013, 11:52 PM | #1 |
Registered User
Join Date: Mar 2013
Posts: 156
|
Robots txt
What is the role of robots file..?
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-03-2013, 05:37 AM | #2 |
Registered User
Join Date: Oct 2012
Posts: 149
|
If you don't want to show any folder , pictures particular any thing than you have to use code .
user-agent* Disalllow http://www.snapdeal.com/thumbals
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-04-2013, 02:35 AM | #3 |
Registered User
Join Date: Jun 2013
Posts: 36
|
If you don't want to cache index any page to search engine crawler then your can use robot.txt
|
07-04-2013, 11:44 PM | #4 |
Registered User
Join Date: Dec 2012
Posts: 657
|
Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-05-2013, 01:10 AM | #5 |
Registered User
Join Date: Mar 2013
Posts: 246
|
Robot.txt is a simple txt file which are use for controlling Search engine's bot to crawl specific pages. you can be restricted search engine bot to crawl the any page by using robot.txt.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-05-2013, 01:44 AM | #6 |
Registered User
Join Date: Jun 2013
Posts: 146
|
Robot.txt is used for hiding the website page from search engines.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-05-2013, 04:17 AM | #7 |
Registered User
Join Date: Jun 2013
Posts: 110
|
Through Robot.txt file we can hide unwated pages of our webiste at the time of crawling.
|
07-05-2013, 05:32 AM | #8 |
Registered User
Join Date: Mar 2013
Posts: 58
|
Robot.txt is a text file that allow or dis allow crawlers to go through or not.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-08-2013, 02:07 PM | #9 |
Registered User
Join Date: Jul 2013
Location: chennai
Posts: 3
|
it helps to hide the secure webpage.
__________________
Tag Technologies - To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. provider |
07-08-2013, 10:35 PM | #10 |
Registered User
Join Date: May 2013
Posts: 115
|
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. || To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-08-2013, 11:23 PM | #11 |
Registered User
Join Date: Jun 2013
Posts: 146
|
If you dont want to show your website pages then use robot.txt.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-09-2013, 12:41 AM | #12 |
Registered User
Join Date: Apr 2012
Location: Bangalore, India
Posts: 6
|
Robots text will prevent crawling the some pages which you don't want to show
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
07-09-2013, 02:24 AM | #13 |
Registered User
Join Date: Jul 2013
Location: Chennai
Posts: 7
|
Robot.txt is a text file that tells the search engine which pages should not crawl and index by search engine in that domain
__________________
Team Mango Media - Leading To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. helps to double your website traffic rate |
07-11-2013, 02:34 AM | #14 |
Registered User
Join Date: Jul 2013
Posts: 40
|
Robot.txt help your page not cache index to search engine.
|
12-19-2016, 01:26 AM | #15 |
Registered User
Join Date: Sep 2016
Posts: 206
|
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Check the website robots file | china-jewels | Bing | 4 | 10-11-2015 08:14 AM |
Uses of robots? | jamesranatte | General Discussion | 2 | 12-11-2014 08:32 AM |
Robots exclusion protocols? | jamesranatte | General Discussion | 7 | 12-26-2013 12:01 AM |
what is difference between spider, crawler and robots? | noahwilson | Search Engine Optimization | 0 | 05-30-2013 03:04 AM |
What is Robots meta tag? | samlko | Search Engine Optimization | 2 | 04-06-2012 12:10 AM |