What is the use of Robots.txt? - Site Owners Forums

jackthomas087 · 02-12-2016, 11:19 PM

Hello friends,

What is the use of Robots.txt? Is it useful when it comes to SEO

Please suggest me about this

Thanks in advance

ahmdtv · 02-13-2016, 01:33 AM

Please click on

AshokDixit89 · 02-16-2016, 12:34 AM

The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

-Robots.txt Allowing Access to Specific Crawlers.
-Allow everything apart from certain patterns of URLs.

RH-Calvin · 02-18-2016, 12:59 AM

Robots.txt is a text file that is inserted into your website to contain instructions for search engine robots. The file lists webpages which are allowed and disallowed from search engine crawling.

Enamul Hoq · 02-20-2016, 07:13 AM

Quote:

Originally Posted by RH-Calvin

Robots.txt is a text file that is inserted into your website to contain instructions for search engine robots. The file lists webpages which are allowed and disallowed from search engine crawling.

I"m also agree with you.

nomi913 · 02-20-2016, 10:06 AM

Robots.txt is a text file that is inserted into your website to contain instructions for search engine robots.

pennyalgarin · 02-26-2016, 07:51 AM

In simple word Robots.txt is one type of text file

Admyrin · 03-01-2016, 05:57 AM

Yes it is usefull, its a text file

tenso500 · 03-02-2016, 05:06 AM

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

Admyrin · 03-04-2016, 03:18 AM

Robots.txt is a text file that is inserted into your website to contain instructions for search engine robots.

andsuma · 04-07-2016, 09:25 PM

Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

rahul3214 · 04-07-2016, 09:38 PM

Robots.txt file is text file which inserted in your web hosting file.
Robots.txt file is important for crawling of your webpage. which page you want to crawl or which not.

Ajaysharma · 02-13-2017, 04:58 AM

Robots.txt is a text file. It is through this file, it gives instruction to search engine crawlers about indexing and caching of a web page, file of a website or directory, domain.

Dennydevi · 02-13-2017, 10:36 PM

It is one of the text file.

jasonroy21 · 02-14-2017, 03:03 AM

Robots.txt Allowing Access to Specific Crawlers.
Allow everything apart from certain patterns of URLs.

02-12-2016, 11:19 PM	#1
jackthomas087 Registered User Join Date: Feb 2014 Location: india Posts: 268	What is the use of Robots.txt? Hello friends, What is the use of Robots.txt? Is it useful when it comes to SEO Please suggest me about this Thanks in advance __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-13-2016, 01:33 AM	#2
ahmdtv Registered User Join Date: Feb 2016 Posts: 18	Please click on __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-16-2016, 12:34 AM	#3
AshokDixit89 Registered User Join Date: Jan 2016 Location: Mumbai, India Posts: 1,064	The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information. -Robots.txt Allowing Access to Specific Crawlers. -Allow everything apart from certain patterns of URLs. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-18-2016, 12:59 AM	#4
RH-Calvin Registered User Join Date: May 2015 Location: Forum Posts: 2,517	Robots.txt is a text file that is inserted into your website to contain instructions for search engine robots. The file lists webpages which are allowed and disallowed from search engine crawling. __________________ █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| $1 VPS Hosting █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Windows Server with Remote Desktop Access █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Servers with Free IPMI Setup

02-20-2016, 10:06 AM	#6
nomi913 Registered User Join Date: Jan 2016 Location: karachi Posts: 38	Robots.txt is a text file that is inserted into your website to contain instructions for search engine robots. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-26-2016, 07:51 AM	#7
pennyalgarin Registered User Join Date: Feb 2016 Posts: 9	Robots.txt In simple word Robots.txt is one type of text file __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-01-2016, 05:57 AM	#8
Admyrin Registered User Join Date: Feb 2016 Location: Surat Posts: 396	Yes it is usefull, its a text file __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-02-2016, 05:06 AM	#9
tenso500 Registered User Join Date: Feb 2016 Posts: 51	The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-04-2016, 03:18 AM	#10
Admyrin Registered User Join Date: Feb 2016 Location: Surat Posts: 396	Robots.txt is a text file that is inserted into your website to contain instructions for search engine robots. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-07-2016, 09:25 PM	#11
andsuma Registered User Join Date: Apr 2016 Posts: 81	Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-07-2016, 09:38 PM	#12
rahul3214 Registered User Join Date: Mar 2016 Posts: 75	Robots.txt file is text file which inserted in your web hosting file. Robots.txt file is important for crawling of your webpage. which page you want to crawl or which not. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-13-2017, 04:58 AM	#13
Ajaysharma Registered User Join Date: Nov 2016 Posts: 372	Robots.txt is a text file. It is through this file, it gives instruction to search engine crawlers about indexing and caching of a web page, file of a website or directory, domain. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-13-2017, 10:36 PM	#14
Dennydevi Registered User Join Date: Sep 2016 Posts: 429	It is one of the text file.

02-14-2017, 03:03 AM	#15
jasonroy21 Registered User Join Date: Jul 2016 Posts: 755	Robots.txt Allowing Access to Specific Crawlers. Allow everything apart from certain patterns of URLs. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM