03-14-2013, 03:48 AM | #1 |
Registered User
Join Date: Nov 2012
Posts: 7
|
Importance of robots.txt in SEO
whats the main benefit of creating a robots.txt file, and how its helps us in seo.
Website Developer India. Ekarigar.com |
03-14-2013, 08:40 AM | #2 |
Registered User
|
this is to prevent bots from indexing certain pages that you do not want people to see. such as your login page, archive pages, backend pages, user pages, etc...also, to prevent indexing of tag and categories
__________________
Great seo company To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. in Florida |
03-14-2013, 10:14 PM | #3 |
Registered User
Join Date: Mar 2013
Posts: 15
|
Generate effective robots.txt files that help ensure Google and other search engines are crawling and indexing your site properly. Robots.txt files create in Noteped ( robots.txt )
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-14-2013, 11:39 PM | #4 |
Registered User
Join Date: Jun 2012
Posts: 239
|
If your site contain such page that you want to index in search engine in that case you add robot text to stop the crawler to visit that and also stop to index that page.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. :: To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-14-2013, 11:51 PM | #5 |
Registered User
Join Date: Feb 2013
Location: Dallas
Posts: 27
|
Robots.txt is an important file if you want to have control on search engine crawlers for your website. If you don't want to show some web pages of your website then robots.txt file will help you in this regard.
__________________
Increase web presence and visibility of your online or local business to reach targeted audience from To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-15-2013, 03:49 AM | #6 |
Registered User
Join Date: Mar 2013
Posts: 246
|
Really robot.txt file is so important for website, its use for control search engine bot to read some private page or some other content, if you want that the visitor not see this page so you can be make it possible with robot.txt.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-15-2013, 05:29 AM | #7 |
Registered User
Join Date: Mar 2013
Location: US
Posts: 12
|
In simple term it gives a massage to the search engine crawlers or spiders which web pages of your site should be indexed and which one should be ignored.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-18-2013, 08:49 PM | #8 |
Registered User
Join Date: Mar 2013
Posts: 3
|
With robot.txt you can control which page google should index and which page it should not crawl.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-18-2013, 08:56 PM | #9 |
Registered User
Join Date: Mar 2013
Posts: 15
|
search engine bot assign limited bandwidth to index a site, robots text can effectively help the bot decide the important page to index.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Malaysia Fair To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Rekasetia To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Alt To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Seo To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Talented |
03-19-2013, 03:20 PM | #10 |
Registered User
Join Date: Dec 2012
Location: Delhi
Posts: 535
|
Robot.txt tell to Google which page should be crawl and which should be not crawl. It is good for website.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. | To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-20-2013, 04:47 AM | #11 |
Registered User
Join Date: May 2012
Posts: 271
|
Robots.txt files inform search engine spiders how to interact with indexing your content.
By default search engines are greedy. They want to index as much high quality information as they can, and will assume that they can crawl everything unless you tell them otherwise. If you specify data for all bots and data for a specific bot (like GoogleBot) then the specific bot commands.
__________________
junk removal san francisco To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-20-2013, 08:35 PM | #12 |
Registered User
Join Date: Mar 2002
Location: Las Vegas, Nevada
Posts: 211
|
interesting can you just find a robot.txt online and modify it?
__________________
Will To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-20-2013, 11:50 PM | #13 |
Registered User
Join Date: Feb 2013
Location: delhi
Posts: 103
|
robots.txt file is very important to crawl or not crawls the pages. as we give coding
User-Agent: * Disallow: if you want to disallow the page disallow:/ event management companies in kolkata
__________________
camp in kanatal |
03-21-2013, 02:47 AM | #14 |
Registered User
Join Date: Mar 2013
Posts: 12
|
Robots.txt is an important file in which you give instructions to search engine crawler that which pages you want to crawl and which are not.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
03-21-2013, 03:33 AM | #15 |
Registered User
Join Date: Nov 2012
Location: Pakistan
Posts: 60
|
Robot.txt is very important as it tells search engine about which folders and pages to index and which not to index.
__________________
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt | Fking | General Discussion | 1 | 12-13-2013 11:27 PM |
How to edit virtual robots.txt file in wordpress? | geniusoptimizer | PHP / mySQL | 1 | 04-12-2013 12:36 PM |
The robots.txt file | rubberfender | Yahoo | 0 | 12-02-2012 07:32 PM |
Does Anyone Know About The Importance Of SEO? | brettlee585 | Search Engine Optimization | 5 | 11-26-2012 05:15 AM |
Unable to delete robots.txt file | notfake | Yahoo | 0 | 04-29-2012 02:19 AM |