What is robot.txt file and what are the benefits of using it? - Page 2 - Site Owners Forums

henrysmithh71 · 11-28-2016, 11:26 PM

Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.

georgewills · 11-28-2016, 11:40 PM

The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.

stuartsmithh5 · 11-28-2016, 11:45 PM

Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.

pattroderick · 11-28-2016, 11:51 PM

Robots.txt file is for giving instruction to the crawlers indicates which one to crawl and which are don't crawl.

Dentcare · 11-29-2016, 04:10 AM

Robot.txt used to provide guidance to the search engine crawler about the web pages that which page to be crawled and which is not.

luffy268 · 12-14-2016, 12:43 AM

The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way. So, if you don't put robots.txt in the right place, do not be surprised that search engines index your whole site.

ChrisRogers123 · 12-14-2016, 02:44 AM

It is a type of text file which allows search engine to what to crawl or what to not

Pandith Raghave · 12-14-2016, 03:13 AM

Robots.txt is document you put on your site to tell look robots which pages you might want them Do and Don't to visit.

jeffmccoy · 12-14-2016, 04:14 AM

Robots.txt file is a file through which we give instructions to the search engines whether to crawl or not the particular web page.

stuartsmithh5 · 12-15-2016, 12:54 AM

Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which are not to be crawled.

Markco92 · 12-15-2016, 03:35 AM

Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not.

FerinKings · 12-16-2016, 12:23 AM

robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

wiliamjamesh · 12-17-2016, 01:49 AM

The purpose of a robots.txt file is to tell search engines not to crawl the contents of specific directories or even individual pages within a website. ... However there are a couple of reasons why all websites would benefit from having one. The first thing search engines look for is the robots.txt file.

georgewills · 12-17-2016, 05:22 AM

Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing.

LukeBarter · 12-19-2016, 01:44 AM

It is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

11-28-2016, 11:26 PM	#16
henrysmithh71 Registered User Join Date: Sep 2016 Posts: 75	Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-28-2016, 11:40 PM	#17
georgewills Registered User Join Date: Sep 2016 Posts: 70	The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-28-2016, 11:45 PM	#18
stuartsmithh5 Registered User Join Date: Oct 2016 Posts: 81	Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-28-2016, 11:51 PM	#19
pattroderick Registered User Join Date: Jul 2016 Posts: 243	Robots.txt file is for giving instruction to the crawlers indicates which one to crawl and which are don't crawl. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

11-29-2016, 04:10 AM	#20
Dentcare Registered User Join Date: Aug 2016 Posts: 187	Robot.txt used to provide guidance to the search engine crawler about the web pages that which page to be crawled and which is not. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-14-2016, 12:43 AM	#21
luffy268 Registered User Join Date: Dec 2016 Posts: 94	The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt. Instead, they look first in the main directory and if they don't find it there, they simply assume that this site does not have a robots.txt file and therefore they index everything they find along the way. So, if you don't put robots.txt in the right place, do not be surprised that search engines index your whole site. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

12-14-2016, 02:44 AM	#22
ChrisRogers123 Registered User Join Date: Aug 2016 Location: USA Posts: 117	It is a type of text file which allows search engine to what to crawl or what to not __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \|\| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-14-2016, 03:13 AM	#23
Pandith Raghave Registered User Join Date: Aug 2016 Location: London Posts: 115	Robots.txt is document you put on your site to tell look robots which pages you might want them Do and Don't to visit. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-14-2016, 04:14 AM	#24
jeffmccoy Registered User Join Date: Feb 2016 Location: Green Cove Springs, FL Posts: 321	Robots.txt file is a file through which we give instructions to the search engines whether to crawl or not the particular web page. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-15-2016, 12:54 AM	#25
stuartsmithh5 Registered User Join Date: Oct 2016 Posts: 81	Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which are not to be crawled. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-15-2016, 03:35 AM	#26
Markco92 Registered User Join Date: Dec 2016 Location: New York Posts: 113	Robot.txt file is used to provide instruction to the search engine crawler about the webpages that which page to be crawled and which not. __________________ Claim ME now anytime to get To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \|\| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \|\| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \|\| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-16-2016, 12:23 AM	#27
FerinKings Registered User Join Date: Aug 2016 Posts: 205	robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

12-17-2016, 01:49 AM	#28
wiliamjamesh Registered User Join Date: Sep 2016 Posts: 617	The purpose of a robots.txt file is to tell search engines not to crawl the contents of specific directories or even individual pages within a website. ... However there are a couple of reasons why all websites would benefit from having one. The first thing search engines look for is the robots.txt file. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-17-2016, 05:22 AM	#29
georgewills Registered User Join Date: Sep 2016 Posts: 70	Robots.txt file is used to give instruction to the search engine crawler to which page has to crawled or not by allowing or disallowing. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-19-2016, 01:44 AM	#30
LukeBarter Registered User Join Date: Sep 2016 Posts: 171	It is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \|\| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)