12-23-2014, 05:43 AM | #1 |
Registered User
Join Date: Oct 2013
Posts: 37
|
Robot.txt?
What is Robot.txt? Why it is used?
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-23-2014, 01:13 PM | #2 |
Registered User
Join Date: Dec 2014
Location: Pakistan
Posts: 16
|
robots.txt file used as directives to search engine crawlers. you can forbid crawlers to crawl specific pages of your site
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-24-2014, 02:01 AM | #3 |
Registered User
Join Date: Jul 2014
Location: bangalore, india
Posts: 66
|
Robots.txt is a text (not html) file you put on your web page to tell search robots which pages you would like them not to visit.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-24-2014, 06:08 AM | #4 |
Registered User
Join Date: Dec 2014
Posts: 31
|
Yes all answers are some but not exact :-
Exactly robots.txt is a text file which has two parts user agent and disallow In dis allow, we place that file we don't want to crawl and index in search engine. |
12-24-2014, 08:42 AM | #5 |
Registered User
Join Date: Dec 2014
Posts: 58
|
I think they explained you why you should keep it pn your host
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-24-2014, 10:56 PM | #6 |
Registered User
Join Date: Sep 2013
Location: Delhi
Posts: 344
|
robots.txt is helpful to crawls the site or not by using allow or disallow.
picnic spots near delhi | New year packages delhi 2015 | camp in kanatal Last edited by rajivdevkk; 12-24-2014 at 11:00 PM.. |
12-26-2014, 10:37 PM | #7 |
Registered User
Join Date: Dec 2014
Posts: 22
|
This helps to prevent a particular page to crowl that the user don't want.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-27-2014, 12:08 AM | #8 |
Registered User
Join Date: Nov 2014
Posts: 12
|
Robots.txt is an text file used by search crewl.it tells crewl which part of website to index or which not index in search result.Find More At Custom Robot.txt
|
12-27-2014, 02:51 AM | #9 |
Registered User
Join Date: Aug 2014
Posts: 98
|
It is used to allow or disallow search engine to crawl your pages..
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-27-2014, 03:00 AM | #10 |
Registered User
Join Date: Apr 2014
Posts: 3
|
It help to instruct crawler that which page should be cache on which is not.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
12-27-2014, 03:51 AM | #11 |
Registered User
Join Date: Apr 2014
Posts: 110
|
Robot.txt file used to block crawler to crawl some content in your website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
01-08-2015, 11:36 PM | #12 |
Registered User
Join Date: Jan 2015
Posts: 6
|
Robots.txt is a text file, It is used to allow or disallow search engines to crawl your website or webpage
|
01-09-2015, 12:38 AM | #13 |
Registered User
Join Date: Jan 2015
Posts: 80
|
Generate effective robots.txt files that help ensure Google and other search engines are crawling and indexing your site properly.
|
01-09-2015, 01:06 AM | #14 |
Registered User
Join Date: Jan 2015
Posts: 1
|
Robots.txt is a text file which we upload on website server to instruct spider the which page or folder should be crawl and which is not.
|
01-09-2015, 01:18 AM | #15 |
Registered User
Join Date: Mar 2014
Posts: 111
|
Thanks for your question and happy to read all friends suggestion
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
what is robot.txt? | rubysethi | Search Engine Optimization | 0 | 12-19-2014 11:27 PM |
How to know about robot.txt ? | vanaja | Search Engine Optimization | 10 | 10-08-2014 04:39 AM |
how can Create the Robot.txt | pankaj | Search Engine Optimization | 3 | 10-08-2014 03:43 AM |
how can create the Robot.txt file | eagleeye dental | Search Engine Optimization | 0 | 09-01-2014 09:40 PM |
how to use robot.txt? | dean1122 | Search Engine Optimization | 11 | 01-05-2013 02:10 AM |