Robots txt - Site Owners Forums

discussion123 · 07-02-2013, 11:52 PM

What is the role of robots file..?

likit · 07-03-2013, 05:37 AM

If you don't want to show any folder , pictures particular any thing than you have to use code .

user-agent*
Disalllow http://www.snapdeal.com/thumbals

astroavl9 · 07-04-2013, 02:35 AM

If you don't want to cache index any page to search engine crawler then your can use robot.txt

jaysh4922 · 07-04-2013, 11:44 PM

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

noahwilson · 07-05-2013, 01:10 AM

Robot.txt is a simple txt file which are use for controlling Search engine's bot to crawl specific pages. you can be restricted search engine bot to crawl the any page by using robot.txt.

seovikrama2z · 07-05-2013, 01:44 AM

Robot.txt is used for hiding the website page from search engines.

vantrendin · 07-05-2013, 04:17 AM

Through Robot.txt file we can hide unwated pages of our webiste at the time of crawling.

jeffzeilenga · 07-05-2013, 05:32 AM

Robot.txt is a text file that allow or dis allow crawlers to go through or not.

laurachrismarti · 07-08-2013, 02:07 PM

it helps to hide the secure webpage.

skytours · 07-08-2013, 10:35 PM

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site

seovikrama2z · 07-08-2013, 11:23 PM

If you dont want to show your website pages then use robot.txt.

soppnox · 07-09-2013, 12:41 AM

Robots text will prevent crawling the some pages which you don't want to show

OliverHenry · 07-09-2013, 02:24 AM

Robot.txt is a text file that tells the search engine which pages should not crawl and index by search engine in that domain

bluelizard95 · 07-11-2013, 02:34 AM

Robot.txt help your page not cache index to search engine.

Jadanjesse · 12-19-2016, 01:26 AM

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site

07-02-2013, 11:52 PM	#1
discussion123 Registered User Join Date: Mar 2013 Posts: 156	Robots txt What is the role of robots file..? __________________ Camp dhanaulti \| Camp O Royale

07-03-2013, 05:37 AM	#2
likit Registered User Join Date: Oct 2012 Posts: 149	If you don't want to show any folder , pictures particular any thing than you have to use code . user-agent* Disalllow http://www.snapdeal.com/thumbals __________________ Horse Supplements

07-04-2013, 11:44 PM	#4
jaysh4922 Registered User Join Date: Dec 2012 Posts: 657	Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content. __________________ Web Development Services SEO Services

07-05-2013, 01:10 AM	#5
noahwilson Registered User Join Date: Mar 2013 Posts: 246	Robot.txt is a simple txt file which are use for controlling Search engine's bot to crawl specific pages. you can be restricted search engine bot to crawl the any page by using robot.txt. __________________ Moving To UK \| Furniture Removalists Perth \| Moving To Europe \| Interstate Removalist

07-05-2013, 01:44 AM	#6
seovikrama2z Registered User Join Date: Jun 2013 Posts: 146	Robot.txt is used for hiding the website page from search engines. __________________ gmail tech support \| gmail helpline number \| gmail toll free number

07-04-2013, 02:35 AM	#3
astroavl9 Registered User Join Date: Jun 2013 Posts: 36	If you don't want to cache index any page to search engine crawler then your can use robot.txt

07-05-2013, 04:17 AM	#7
vantrendin Registered User Join Date: Jun 2013 Posts: 110	Through Robot.txt file we can hide unwated pages of our webiste at the time of crawling.

07-05-2013, 05:32 AM	#8
jeffzeilenga Registered User Join Date: Mar 2013 Posts: 58	Robot.txt is a text file that allow or dis allow crawlers to go through or not. __________________ ball valve manufacturer

07-08-2013, 02:07 PM	#9
laurachrismarti Registered User Join Date: Jul 2013 Location: chennai Posts: 3	it helps to hide the secure webpage. __________________ Tag Technologies - professional web design & web development solutions provider

07-08-2013, 10:35 PM	#10
skytours Registered User Join Date: May 2013 Posts: 115	Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site __________________ Cheap Airline Tickets \|\| Cheap Airfares

07-08-2013, 11:23 PM	#11
seovikrama2z Registered User Join Date: Jun 2013 Posts: 146	If you dont want to show your website pages then use robot.txt. __________________ gmail tech support \| gmail helpline number \| gmail toll free number

07-09-2013, 12:41 AM	#12
soppnox Registered User Join Date: Apr 2012 Location: Bangalore, India Posts: 6	Robots text will prevent crawling the some pages which you don't want to show __________________ Search Engine Optimisation Best Web Design Company Internet Marketing in India SEO Company India Best SEO Company

07-09-2013, 02:24 AM	#13
OliverHenry Registered User Join Date: Jul 2013 Location: Chennai Posts: 7	Robot.txt is a text file that tells the search engine which pages should not crawl and index by search engine in that domain __________________ Team Mango Media - Leading Seo Company in India helps to double your website traffic rate

07-11-2013, 02:34 AM	#14
bluelizard95 Registered User Join Date: Jul 2013 Posts: 40	Robot.txt help your page not cache index to search engine.

12-19-2016, 01:26 AM	#15
Jadanjesse Registered User Join Date: Sep 2016 Posts: 206	Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site __________________ Data Conversion Archives Medical Data Conversions

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Check the website robots file	china-jewels	Bing	4	10-11-2015 08:14 AM
Uses of robots?	jamesranatte	General Discussion	2	12-11-2014 08:32 AM
Robots exclusion protocols?	jamesranatte	General Discussion	7	12-26-2013 12:01 AM
what is difference between spider, crawler and robots?	noahwilson	Search Engine Optimization	0	05-30-2013 03:04 AM
What is Robots meta tag?	samlko	Search Engine Optimization	2	04-06-2012 12:10 AM