Importance of robots.txt in SEO - Site Owners Forums

TaylorZenith · 03-14-2013, 03:48 AM

whats the main benefit of creating a robots.txt file, and how its helps us in seo.
Website Developer India. Ekarigar.com

joshcooper · 03-14-2013, 08:40 AM

this is to prevent bots from indexing certain pages that you do not want people to see. such as your login page, archive pages, backend pages, user pages, etc...also, to prevent indexing of tag and categories

cloudsmoking · 03-14-2013, 10:14 PM

Generate effective robots.txt files that help ensure Google and other search engines are crawling and indexing your site properly. Robots.txt files create in Noteped ( robots.txt )

JACKADEN · 03-14-2013, 11:39 PM

If your site contain such page that you want to index in search engine in that case you add robot text to stop the crawler to visit that and also stop to index that page.

socialmediadfw · 03-14-2013, 11:51 PM

Robots.txt is an important file if you want to have control on search engine crawlers for your website. If you don't want to show some web pages of your website then robots.txt file will help you in this regard.

noahwilson · 03-15-2013, 03:49 AM

Really robot.txt file is so important for website, its use for control search engine bot to read some private page or some other content, if you want that the visitor not see this page so you can be make it possible with robot.txt.

beatrixtuffy · 03-15-2013, 05:29 AM

In simple term it gives a massage to the search engine crawlers or spiders which web pages of your site should be indexed and which one should be ignored.

vikash_munda345 · 03-18-2013, 08:49 PM

With robot.txt you can control which page google should index and which page it should not crawl.

altcom · 03-18-2013, 08:56 PM

search engine bot assign limited bandwidth to index a site, robots text can effectively help the bot decide the important page to index.

Kunalmathur · 03-19-2013, 03:20 PM

Robot.txt tell to Google which page should be crawl and which should be not crawl. It is good for website.

josephjohn4556 · 03-20-2013, 04:47 AM

Robots.txt files inform search engine spiders how to interact with indexing your content.
By default search engines are greedy. They want to index as much high quality information as they can, and will assume that they can crawl everything unless you tell them otherwise.
If you specify data for all bots and data for a specific bot (like GoogleBot) then the specific bot commands.

Budd · 03-20-2013, 08:35 PM

interesting can you just find a robot.txt online and modify it?

danish00 · 03-20-2013, 11:50 PM

robots.txt file is very important to crawl or not crawls the pages. as we give coding
User-Agent: *
Disallow:
if you want to disallow the page
disallow:/
event management companies in kolkata

richas9999 · 03-21-2013, 02:47 AM

Robots.txt is an important file in which you give instructions to search engine crawler that which pages you want to crawl and which are not.

tanveerhabib · 03-21-2013, 03:33 AM

Robot.txt is very important as it tells search engine about which folders and pages to index and which not to index.

03-14-2013, 03:48 AM	#1
TaylorZenith Registered User Join Date: Nov 2012 Posts: 7	Importance of robots.txt in SEO whats the main benefit of creating a robots.txt file, and how its helps us in seo. Website Developer India. Ekarigar.com

03-14-2013, 08:40 AM	#2
joshcooper Registered User Join Date: Mar 2013 Location: Florida Posts: 10	this is to prevent bots from indexing certain pages that you do not want people to see. such as your login page, archive pages, backend pages, user pages, etc...also, to prevent indexing of tag and categories __________________ Great seo company To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. in Florida

03-14-2013, 10:14 PM	#3
cloudsmoking Registered User Join Date: Mar 2013 Posts: 15	Generate effective robots.txt files that help ensure Google and other search engines are crawling and indexing your site properly. Robots.txt files create in Noteped ( robots.txt ) __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-14-2013, 11:39 PM	#4
JACKADEN Registered User Join Date: Jun 2012 Posts: 239	If your site contain such page that you want to index in search engine in that case you add robot text to stop the crawler to visit that and also stop to index that page. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. :: To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-14-2013, 11:51 PM	#5
socialmediadfw Registered User Join Date: Feb 2013 Location: Dallas Posts: 27	Robots.txt is an important file if you want to have control on search engine crawlers for your website. If you don't want to show some web pages of your website then robots.txt file will help you in this regard. __________________ Increase web presence and visibility of your online or local business to reach targeted audience from *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

03-15-2013, 03:49 AM	#6
noahwilson Registered User Join Date: Mar 2013 Posts: 246	Really robot.txt file is so important for website, its use for control search engine bot to read some private page or some other content, if you want that the visitor not see this page so you can be make it possible with robot.txt. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-15-2013, 05:29 AM	#7
beatrixtuffy Registered User Join Date: Mar 2013 Location: US Posts: 12	In simple term it gives a massage to the search engine crawlers or spiders which web pages of your site should be indexed and which one should be ignored. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-18-2013, 08:49 PM	#8
vikash_munda345 Registered User Join Date: Mar 2013 Posts: 3	With robot.txt you can control which page google should index and which page it should not crawl. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-18-2013, 08:56 PM	#9
altcom Registered User Join Date: Mar 2013 Posts: 15	search engine bot assign limited bandwidth to index a site, robots text can effectively help the bot decide the important page to index. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Malaysia Fair To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Rekasetia To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Alt To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Seo To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Talented

03-19-2013, 03:20 PM	#10
Kunalmathur Registered User Join Date: Dec 2012 Location: Delhi Posts: 535	Robot.txt tell to Google which page should be crawl and which should be not crawl. It is good for website. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

03-20-2013, 04:47 AM	#11
josephjohn4556 Registered User Join Date: May 2012 Posts: 271	Robots.txt files inform search engine spiders how to interact with indexing your content. By default search engines are greedy. They want to index as much high quality information as they can, and will assume that they can crawl everything unless you tell them otherwise. If you specify data for all bots and data for a specific bot (like GoogleBot) then the specific bot commands. __________________ junk removal san francisco To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-20-2013, 08:35 PM	#12
Budd Registered User Join Date: Mar 2002 Location: Las Vegas, Nevada Posts: 211	interesting can you just find a robot.txt online and modify it? __________________ Will To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-20-2013, 11:50 PM	#13
danish00 Registered User Join Date: Feb 2013 Location: delhi Posts: 103	robots.txt file is very important to crawl or not crawls the pages. as we give coding User-Agent: * Disallow: if you want to disallow the page disallow:/ event management companies in kolkata __________________ camp in kanatal

03-21-2013, 02:47 AM	#14
richas9999 Registered User Join Date: Mar 2013 Posts: 12	Robots.txt is an important file in which you give instructions to search engine crawler that which pages you want to crawl and which are not. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-21-2013, 03:33 AM	#15
tanveerhabib Registered User Join Date: Nov 2012 Location: Pakistan Posts: 60	Robot.txt is very important as it tells search engine about which folders and pages to index and which not to index. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Does Anyone Know About The Importance Of SEO?	brettlee585	Search Engine Optimization	5	11-26-2012 05:15 AM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM