What is robot text
 

Do you know about robot text? please say and discuss about it.

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.Robot.txt is used to prevent crawlers crawl on a certain page.

By the help of robots.txt file, search engines crawls your webpages or not crawls your pages.
day picnic around delhi

Robot.txt is an on-page SEO technique and it is basically used to allow for the web robots also known as the web wanderers, crawlers or spiders. It is a program that traverses the website automatically and this helps the popular search engine like Google to index the website and its content.

Just create a file named robots.txt, put it on your root directory of your website(where index.htm/index.php is), edit with a text editor and add these:

User-agent: *
Disallow: *something*

robots.txt is a text file which instructs the search engine what are the pages should be crawled.

Robots.txt is the text file that is mostly used to instruct search engine which page should be crawled and which shouldn't be crawled.

Robots.txt is a file that you can add to your website which instructs the search engine crawler which pages to crawl. You can also instruct the crawlers not to crawl some pages through robots.txt.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

The structure of a robots.txt is pretty simple (and barely flexible) – it is an endless list of user agents and disallowed files and directories. Basically, the syntax is as follows:

User-agent:

Disallow:

“User-agent” are search engines' crawlers and disallow: lists the files and directories to be excluded from indexing. In addition to “user-agent:” and “disallow:” entries, you can include comment lines – just put the # sign at the beginning of the line:

# All user agents are disallowed to see the /temp directory.

User-agent: *

Disallow: /temp/

Robots.txt file is in the root directory of a website which has been created to direct
the activity of search engine crawlers or spiders.

The concept and structure of robots.txt has been developed more than a decade ago. Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. The location of robots.txt is very important. It must be in the main directory because otherwise user agents (search engines) will not be able to find it – they do not search the whole site for a file named robots.txt.

Go on......Thanks all..

as you can easily judge by its name, robot txt means message or txt for robot (search engine spiders). In you webmaster tell bots which page to crawl or which not.

I do not see anymore effects of Robot.text file. Search Engine also index those one which comes under no follow. I tried that. No effect of this file now.

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.

robots text is normal file in which you can give instruction which page you crawl or not to all robots of search engines.