Importance of Robots.txt File - Site Owners Forums

omkarsoft · 11-21-2014, 12:03 AM

When a search engine crawler involves your web site, it'll look for a special file on your site named robots.txt and it tells the search engine spider, that web pages of your website should be indexed.

GinnySweety · 11-21-2014, 02:31 AM

robots.txt file is very important to rank good in search engine results page. The work of robots.txt file is when search engine crawler visit your website, it mainly search for robots.txt file and it tells the search engine spider, which web page has to be indexed and which page should be avoided.

jeorge · 11-21-2014, 04:58 AM

robots.txt file is a very important term in on-page optimization in SEO. we can give the the instruction to search engine crawler that which is crawled and which page is avoided.

sripriya2691 · 11-21-2014, 06:10 AM

Robots.txt is used to apply condition for bots

William Lacey · 11-22-2014, 11:09 AM

So a website with a robots.txt file being used correctly, will help a website rank higher in Google, is that correct?

brainmoore · 11-22-2014, 11:04 PM

Robots.txt is basically a text file which is designed for machines, robots which crawl websites online. It uses some codes which tells bots whether to crawl the website/page or not, it is more like a permission given area.

teeceepee · 11-24-2014, 09:26 PM

Basically, without it, you're wasting time on other SEO initiatives.

jobtardisport · 11-24-2014, 11:13 PM

A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.

BillieM · 11-25-2014, 04:42 PM

it's one of the fundamental building blocks of good search engine optimisation, so yes it is essential.

SyedZaidi · 11-26-2014, 11:25 PM

Robots.txt informs the crawler what to crawl and what not crawl.

larrypaul · 11-27-2014, 02:17 AM

Robots.txt is an important essential element for making google bot crawl the website and also those pages which we wish not to index, can be made disallowed by robots file...

11-21-2014, 12:03 AM	#1
omkarsoft Registered User Join Date: Sep 2014 Location: Bangalore,Karnataka Posts: 133	Importance of Robots.txt File When a search engine crawler involves your web site, it'll look for a special file on your site named robots.txt and it tells the search engine spider, that web pages of your website should be indexed. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-21-2014, 02:31 AM	#2
GinnySweety Registered User Join Date: Sep 2014 Posts: 84	robots.txt file is very important to rank good in search engine results page. The work of robots.txt file is when search engine crawler visit your website, it mainly search for robots.txt file and it tells the search engine spider, which web page has to be indexed and which page should be avoided. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-21-2014, 04:58 AM	#3
jeorge Registered User Join Date: Nov 2014 Posts: 27	robots.txt file is a very important term in on-page optimization in SEO. we can give the the instruction to search engine crawler that which is crawled and which page is avoided. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-21-2014, 06:10 AM	#4
sripriya2691 Registered User Join Date: Nov 2014 Posts: 1	Robots.txt is used to apply condition for bots __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-22-2014, 11:09 AM	#5
William Lacey Registered User Join Date: Oct 2014 Location: Woking/Surrey Posts: 20	So a website with a robots.txt file being used correctly, will help a website rank higher in Google, is that correct? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-22-2014, 11:04 PM	#6
brainmoore Registered User Join Date: Oct 2014 Posts: 32	Robots.txt is basically a text file which is designed for machines, robots which crawl websites online. It uses some codes which tells bots whether to crawl the website/page or not, it is more like a permission given area. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-24-2014, 09:26 PM	#7
teeceepee Registered User Join Date: Nov 2014 Posts: 10	Basically, without it, you're wasting time on other SEO initiatives. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-24-2014, 11:13 PM	#8
jobtardisport Registered User Join Date: Sep 2013 Posts: 887	A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-25-2014, 04:42 PM	#9
BillieM Registered User Join Date: Nov 2014 Posts: 10	it's one of the fundamental building blocks of good search engine optimisation, so yes it is essential. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-26-2014, 11:25 PM	#10
SyedZaidi Registered User Join Date: Oct 2014 Posts: 41	Robots.txt informs the crawler what to crawl and what not crawl. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-27-2014, 02:17 AM	#11
larrypaul Registered User Join Date: Sep 2014 Posts: 55	Robots.txt is an important essential element for making google bot crawl the website and also those pages which we wish not to index, can be made disallowed by robots file... __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
What is the purpose of robots.txt file?	sahithya	Search Engine Optimization	23	12-28-2021 03:48 AM
What Is A Robots.txt File?	ThedorisJackson	General Discussion	5	07-22-2014 04:28 AM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM