Robot.txt? - Site Owners Forums

siva SEO · 12-23-2014, 05:43 AM

What is Robot.txt? Why it is used?

aliblogger1985 · 12-23-2014, 01:13 PM

robots.txt file used as directives to search engine crawlers. you can forbid crawlers to crawl specific pages of your site

Bestseoservices · 12-24-2014, 02:01 AM

Robots.txt is a text (not html) file you put on your web page to tell search robots which pages you would like them not to visit.

ramphal · 12-24-2014, 06:08 AM

Yes all answers are some but not exact :-
Exactly robots.txt is a text file which has two parts
user agent and disallow
In dis allow, we place that file we don't want to crawl and index in search engine.

User1337 · 12-24-2014, 08:42 AM

I think they explained you why you should keep it pn your host

rajivdevkk · 12-24-2014, 10:56 PM

robots.txt is helpful to crawls the site or not by using allow or disallow.

picnic spots near delhi | New year packages delhi 2015 | camp in kanatal

universeseo5 · 12-26-2014, 10:37 PM

This helps to prevent a particular page to crowl that the user don't want.

boyg600 · 12-27-2014, 12:08 AM

Robots.txt is an text file used by search crewl.it tells crewl which part of website to index or which not index in search result.Find More At Custom Robot.txt

DuryabAziz · 12-27-2014, 02:51 AM

It is used to allow or disallow search engine to crawl your pages..

Aaliyah Liverpo · 12-27-2014, 03:00 AM

It help to instruct crawler that which page should be cache on which is not.

creativesquad · 12-27-2014, 03:51 AM

Robot.txt file used to block crawler to crawl some content in your website.

Cleantekmarket · 01-08-2015, 11:36 PM

Robots.txt is a text file, It is used to allow or disallow search engines to crawl your website or webpage

Manjot · 01-09-2015, 12:38 AM

Generate effective robots.txt files that help ensure Google and other search engines are crawling and indexing your site properly.

stephinewinkle · 01-09-2015, 01:06 AM

Robots.txt is a text file which we upload on website server to instruct spider the which page or folder should be crawl and which is not.

Agrodut · 01-09-2015, 01:18 AM

Thanks for your question and happy to read all friends suggestion

12-23-2014, 05:43 AM	#1
siva SEO Registered User Join Date: Oct 2013 Posts: 37	Robot.txt? What is Robot.txt? Why it is used? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-23-2014, 01:13 PM	#2
aliblogger1985 Registered User Join Date: Dec 2014 Location: Pakistan Posts: 16	robots.txt file used as directives to search engine crawlers. you can forbid crawlers to crawl specific pages of your site __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-24-2014, 02:01 AM	#3
Bestseoservices Registered User Join Date: Jul 2014 Location: bangalore, india Posts: 66	Robots.txt is a text (not html) file you put on your web page to tell search robots which pages you would like them not to visit. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-24-2014, 08:42 AM	#5
User1337 Registered User Join Date: Dec 2014 Posts: 58	I think they explained you why you should keep it pn your host __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-24-2014, 10:56 PM	#6
rajivdevkk Registered User Join Date: Sep 2013 Location: Delhi Posts: 344	robots.txt is helpful to crawls the site or not by using allow or disallow. picnic spots near delhi \| New year packages delhi 2015 \| camp in kanatal Last edited by rajivdevkk; 12-24-2014 at 11:00 PM..

12-24-2014, 06:08 AM	#4
ramphal Registered User Join Date: Dec 2014 Posts: 31	Yes all answers are some but not exact :- Exactly robots.txt is a text file which has two parts user agent and disallow In dis allow, we place that file we don't want to crawl and index in search engine.

12-26-2014, 10:37 PM	#7
universeseo5 Registered User Join Date: Dec 2014 Posts: 22	This helps to prevent a particular page to crowl that the user don't want. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-27-2014, 12:08 AM	#8
boyg600 Registered User Join Date: Nov 2014 Posts: 12	Robots.txt is an text file used by search crewl.it tells crewl which part of website to index or which not index in search result.Find More At Custom Robot.txt

12-27-2014, 02:51 AM	#9
DuryabAziz Registered User Join Date: Aug 2014 Posts: 98	It is used to allow or disallow search engine to crawl your pages.. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-27-2014, 03:00 AM	#10
Aaliyah Liverpo Registered User Join Date: Apr 2014 Posts: 3	It help to instruct crawler that which page should be cache on which is not. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

12-27-2014, 03:51 AM	#11
creativesquad Registered User Join Date: Apr 2014 Posts: 110	Robot.txt file used to block crawler to crawl some content in your website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

01-08-2015, 11:36 PM	#12
Cleantekmarket Registered User Join Date: Jan 2015 Posts: 6	Robots.txt is a text file, It is used to allow or disallow search engines to crawl your website or webpage

01-09-2015, 12:38 AM	#13
Manjot Registered User Join Date: Jan 2015 Posts: 80	Generate effective robots.txt files that help ensure Google and other search engines are crawling and indexing your site properly.

01-09-2015, 01:06 AM	#14
stephinewinkle Registered User Join Date: Jan 2015 Posts: 1	Robots.txt is a text file which we upload on website server to instruct spider the which page or folder should be crawl and which is not.

01-09-2015, 01:18 AM	#15
Agrodut Registered User Join Date: Mar 2014 Posts: 111	Thanks for your question and happy to read all friends suggestion __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
what is robot.txt?	rubysethi	Search Engine Optimization	0	12-19-2014 11:27 PM
How to know about robot.txt ?	vanaja	Search Engine Optimization	10	10-08-2014 04:39 AM
how can Create the Robot.txt	pankaj	Search Engine Optimization	3	10-08-2014 03:43 AM
how can create the Robot.txt file	eagleeye dental	Search Engine Optimization	0	09-01-2014 09:40 PM
how to use robot.txt?	dean1122	Search Engine Optimization	11	01-05-2013 02:10 AM