What is robot text - Site Owners Forums

asad111720 · 02-24-2013, 02:18 AM

Do you know about robot text? please say and discuss about it.

champion · 02-24-2013, 10:20 PM

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.Robot.txt is used to prevent crawlers crawl on a certain page.

danish00 · 02-25-2013, 12:19 AM

By the help of robots.txt file, search engines crawls your webpages or not crawls your pages.
day picnic around delhi

jaysh4922 · 02-25-2013, 03:18 AM

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

Just create a file named robots.txt, put it on your root directory of your website(where index.htm/index.php is), edit with a text editor and add these:

User-agent: *
Disallow: *something*

roger4321 · 02-25-2013, 05:31 AM

robots.txt is a text file which instructs the search engine what are the pages should be crawled.

Zora2012 · 02-25-2013, 05:33 AM

Robots.txt is the text file that is mostly used to instruct search engine which page should be crawled and which shouldn't be crawled.

SamMarley · 02-25-2013, 06:10 AM

Robots.txt is a file that you can add to your website which instructs the search engine crawler which pages to crawl. You can also instruct the crawlers not to crawl some pages through robots.txt.

david0019 · 02-25-2013, 06:23 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

The structure of a robots.txt is pretty simple (and barely flexible) – it is an endless list of user agents and disallowed files and directories. Basically, the syntax is as follows:

User-agent:

Disallow:

“User-agent” are search engines' crawlers and disallow: lists the files and directories to be excluded from indexing. In addition to “user-agent:” and “disallow:” entries, you can include comment lines – just put the # sign at the beginning of the line:

# All user agents are disallowed to see the /temp directory.

User-agent: *

Disallow: /temp/

spaculus · 02-25-2013, 07:08 AM

Robots.txt file is in the root directory of a website which has been created to direct
the activity of search engine crawlers or spiders.

jayanta1 · 02-25-2013, 11:00 PM

The concept and structure of robots.txt has been developed more than a decade ago. Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt.

asad111720 · 02-26-2013, 03:07 PM

Go on......Thanks all..

halenmartinez · 02-27-2013, 03:30 AM

as you can easily judge by its name, robot txt means message or txt for robot (search engine spiders). In you webmaster tell bots which page to crawl or which not.

spaculus · 02-28-2013, 04:12 AM

I do not see anymore effects of Robot.text file. Search Engine also index those one which comes under no follow. I tried that. No effect of this file now.

James Andy · 02-28-2013, 05:16 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

delhicourses · 02-28-2013, 09:12 AM

robots text is normal file in which you can give instruction which page you crawl or not to all robots of search engines.

02-24-2013, 02:18 AM	#1
asad111720 Registered User Join Date: Jan 2013 Posts: 73	What is robot text Do you know about robot text? please say and discuss about it. __________________ AMAZON PROMOTIONAL CODES To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-25-2013, 12:19 AM	#3
danish00 Registered User Join Date: Feb 2013 Location: delhi Posts: 103	By the help of robots.txt file, search engines crawls your webpages or not crawls your pages. day picnic around delhi __________________ camp in kanatal

02-25-2013, 03:18 AM	#4
jaysh4922 Registered User Join Date: Dec 2012 Posts: 657	Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content. Just create a file named robots.txt, put it on your root directory of your website(where index.htm/index.php is), edit with a text editor and add these: User-agent: * Disallow: something __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-25-2013, 05:31 AM	#5
roger4321 Registered User Join Date: Feb 2013 Posts: 165	robots.txt is a text file which instructs the search engine what are the pages should be crawled. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-25-2013, 05:33 AM	#6
Zora2012 Registered User Join Date: Nov 2012 Location: India Posts: 276	Robots.txt is the text file that is mostly used to instruct search engine which page should be crawled and which shouldn't be crawled. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Shopping Cart- PrestaShop, Zencart; CMS - WordPress, Joomla; Custom Development To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - SEO, Pay Per Click (PPC), SEM, Link Popularity Building, SEO Copywriting, SMO, SMM

02-24-2013, 10:20 PM	#2
champion Registered User Join Date: Feb 2013 Posts: 21	Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.Robot.txt is used to prevent crawlers crawl on a certain page.

02-25-2013, 06:10 AM	#7
SamMarley Registered User Join Date: Feb 2013 Posts: 2	Robots.txt is a file that you can add to your website which instructs the search engine crawler which pages to crawl. You can also instruct the crawlers not to crawl some pages through robots.txt.

02-25-2013, 06:23 AM	#8
david0019 Registered User Join Date: Feb 2013 Location: Ahmedabad Posts: 39	Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. The structure of a robots.txt is pretty simple (and barely flexible) – it is an endless list of user agents and disallowed files and directories. Basically, the syntax is as follows: User-agent: Disallow: “User-agent” are search engines' crawlers and disallow: lists the files and directories to be excluded from indexing. In addition to “user-agent:” and “disallow:” entries, you can include comment lines – just put the # sign at the beginning of the line: # All user agents are disallowed to see the /temp directory. User-agent: * Disallow: /temp/

02-25-2013, 07:08 AM	#9
spaculus Registered User Join Date: May 2012 Location: India Posts: 64	Robots.txt file is in the root directory of a website which has been created to direct the activity of search engine crawlers or spiders. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-25-2013, 11:00 PM	#10
jayanta1 Registered User Join Date: Feb 2013 Posts: 44	The concept and structure of robots.txt has been developed more than a decade ago. Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt.

02-26-2013, 03:07 PM	#11
asad111720 Registered User Join Date: Jan 2013 Posts: 73	Go on......Thanks all.. __________________ AMAZON PROMOTIONAL CODES To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-27-2013, 03:30 AM	#12
halenmartinez Registered User Join Date: Feb 2013 Location: Philadelphia USA Posts: 76	as you can easily judge by its name, robot txt means message or txt for robot (search engine spiders). In you webmaster tell bots which page to crawl or which not. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-28-2013, 04:12 AM	#13
spaculus Registered User Join Date: May 2012 Location: India Posts: 64	I do not see anymore effects of Robot.text file. Search Engine also index those one which comes under no follow. I tried that. No effect of this file now. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-28-2013, 05:16 AM	#14
James Andy Registered User Join Date: May 2012 Posts: 65	Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-28-2013, 09:12 AM	#15
delhicourses Registered User Join Date: Dec 2012 Location: delhi Posts: 56	robots text is normal file in which you can give instruction which page you crawl or not to all robots of search engines. __________________ hi i am a webmaster who helps all peoples to get rank well.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
telepresence robot	hemnchy	General Discussion	1	01-02-2014 04:35 AM
Robot text.	terrijhon	Search Engine Optimization	18	06-08-2013 03:35 PM
Hidden text and links	farm gate	Google	0	11-11-2012 08:08 PM
Anchor Text Vs Google Algorithm	Jimerson12	Search Engine Optimization	0	08-16-2012 03:26 AM
Text Areas	ncjbhd	HTML / DHTML	1	11-15-2011 04:09 AM