what is the use of robot.txt? - Page 2 - Site Owners Forums

DuryabAziz · 03-24-2016, 12:56 AM

robot.txt is a file which specifies a search engine bot, if it has to follow your or index your blog or website links or not...

Danbou · 03-24-2016, 01:34 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

spmfacilities · 03-25-2016, 09:14 AM

Robots.txt help to block the crawler to crawl the web page which we mentioned those page URL on robots tag.

pramodkumar · 03-25-2016, 10:56 PM

A Robots.txt file can control the Search Engine bots (eg. Google, bing, Yahoo, Ask, Yandex). If you want to prevent any directory/content of your website from indexing into search. Then using robots.txt you can block those content or directory.

rahul3214 · 04-07-2016, 09:48 PM

Robots.txt is a text file in your web hosting files. Search engine see first, which page crawl or which not

rahul3214 · 04-07-2016, 09:52 PM

In Robots.txt file you can allow or disallow the page you want.
Example:
User-agent : *
Disallow: /
Allow: /

Put your URL in Disallow and allow to crawl or not crawl your page

puneet · 04-07-2016, 09:57 PM

The Robots exclusion protocol(REP) or robots.txt is a text file, search engine spider first see that which page crawl or which not.

Sangeetakashyap · 04-07-2016, 11:22 PM

Robot.txt is the text file that helps the search engine crawler to check your website and its sub pages whether it is good or not. Primarily robot.txt helps the search engine crawlers to index the pages of a website quickly and easily. It is the important factor which a website must contain in it for getting good rank in different search engines.

jamesBA · 04-09-2016, 11:59 AM

here are two important considerations when using /robots.txt:

1- robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
2-the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use

varnajalam · 04-10-2016, 10:11 PM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

To Get More info - SEO Hats | SEO Updates

leenahahim · 04-10-2016, 10:13 PM

Robots.txt file will helps to block all web crawlers from all web pages or specific web crawlers from specific web pages.

orientation · 04-11-2016, 12:35 AM

robots.txt is a text file which is used by web-master to give crawling instructions to the search engine crawlers which page is to be crawl and which page is not be crawled

puneet · 04-11-2016, 01:02 AM

Robots.txt file help you to which web url will show on search engine or which not

kirakira2541 · 04-16-2016, 06:32 AM

robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

03-24-2016, 12:56 AM	#16
DuryabAziz Registered User Join Date: Aug 2014 Posts: 98	robot.txt is a file which specifies a search engine bot, if it has to follow your or index your blog or website links or not... __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-24-2016, 01:34 AM	#17
Danbou Registered User Join Date: Jan 2016 Location: New york Posts: 186	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds: __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Taste Of The Wild Dog To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. – WORK VIA THE INTERNET WITH To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-25-2016, 09:14 AM	#18
spmfacilities Registered User Join Date: Feb 2016 Location: India Posts: 21	Robots.txt help to block the crawler to crawl the web page which we mentioned those page URL on robots tag. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-25-2016, 10:56 PM	#19
pramodkumar Registered User Join Date: Mar 2016 Location: Etawah, India Posts: 25	A Robots.txt file can control the Search Engine bots (eg. Google, bing, Yahoo, Ask, Yandex). If you want to prevent any directory/content of your website from indexing into search. Then using robots.txt you can block those content or directory. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-07-2016, 09:48 PM	#20
rahul3214 Registered User Join Date: Mar 2016 Posts: 75	Robots.txt is a text file in your web hosting files. Search engine see first, which page crawl or which not __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-07-2016, 09:52 PM	#21
rahul3214 Registered User Join Date: Mar 2016 Posts: 75	In Robots.txt file you can allow or disallow the page you want. Example: User-agent : * Disallow: / Allow: / Put your URL in Disallow and allow to crawl or not crawl your page __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-07-2016, 09:57 PM	#22
puneet Registered User Join Date: Apr 2016 Posts: 26	The Robots exclusion protocol(REP) or robots.txt is a text file, search engine spider first see that which page crawl or which not.

04-07-2016, 11:22 PM	#23
Sangeetakashyap Registered User Join Date: Sep 2015 Location: Delhi Posts: 145	Robot.txt is the text file that helps the search engine crawler to check your website and its sub pages whether it is good or not. Primarily robot.txt helps the search engine crawlers to index the pages of a website quickly and easily. It is the important factor which a website must contain in it for getting good rank in different search engines. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-09-2016, 11:59 AM	#24
jamesBA Registered User Join Date: Apr 2016 Posts: 14	here are two important considerations when using /robots.txt: 1- robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention. 2-the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use __________________ JBAbusiness45 - James.Benjamin.Anderson

04-10-2016, 10:11 PM	#25
varnajalam Registered User Join Date: Feb 2016 Posts: 18	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. To Get More info - SEO Hats \| SEO Updates __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \|

04-10-2016, 10:13 PM	#26
leenahahim Registered User Join Date: Feb 2016 Posts: 454	Robots.txt file will helps to block all web crawlers from all web pages or specific web crawlers from specific web pages.

04-11-2016, 12:35 AM	#27
orientation Registered User Join Date: Mar 2016 Location: bangalore Posts: 75	robots.txt is a text file which is used by web-master to give crawling instructions to the search engine crawlers which page is to be crawl and which page is not be crawled __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-11-2016, 01:02 AM	#28
puneet Registered User Join Date: Apr 2016 Posts: 26	Robots.txt file help you to which web url will show on search engine or which not __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-16-2016, 06:32 AM	#29
kirakira2541 Registered User Join Date: Mar 2016 Posts: 158	robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
what is robot.txt?	rubysethi	Search Engine Optimization	0	12-19-2014 11:27 PM
How to know about robot.txt ?	vanaja	Search Engine Optimization	10	10-08-2014 04:39 AM
how can Create the Robot.txt	pankaj	Search Engine Optimization	3	10-08-2014 03:43 AM
how can create the Robot.txt file	eagleeye dental	Search Engine Optimization	0	09-01-2014 09:40 PM