What is robot text - Page 2 - Site Owners Forums

executiveedge01 · 02-28-2013, 05:26 PM

Robots.txt is only using for security purpose.It will not allow an anonymous users frequently into the website.

synchronous · 02-28-2013, 11:02 PM

Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone.

sijugk · 03-04-2013, 08:29 AM

Robots .txt file is the place where you can write rules for web spiders about how to crawl the site and which pages should be excluded. You can write rules specifically for each web spider.

aartikrypton · 03-05-2013, 04:03 AM

Hi,

With the help of robots.txt file search engines crawls your webpages.......

Thanks
Team from
kryptonsoft

jesse12 · 03-05-2013, 04:18 AM

Robots file is a simple text file which is uploaded on root directory this is the file which is crawled by the crawler very first. So in this file we write the code which file we want to block from the crawler and other one we want to crawl.

Main use of this file is just to block the admin section which generally we wish do not crawl by the search engine.

Hope you understood.

aartikrypton · 03-05-2013, 04:36 AM

Hello,

With the help of robots.txt file search engines crawls your webpages.......

Thanks
Team from
Kryptonsoft

iPhone App Development
iPhone Games Development
Offshore iPhone Development

tech.biztech · 03-05-2013, 04:43 AM

It is a text file which tells to search engine robots that which web page you would not like to visit. It is one kind of file or folder which tells search engine to avoid the operations described in robot.txt file.

JACKADEN · 03-06-2013, 12:22 AM

Here is some information about Robot text.

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

It works like this: a robot wants to visit a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds:

User-agent: *
Disallow: /

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
There are two important considerations when using /robots.txt:

• Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
• The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

So don't try to use /robots.txt to hide information.

spaculus · 03-06-2013, 06:25 AM

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

thejkamath · 03-07-2013, 06:31 AM

Robot.txt is a file that instructs crawlers and bots whether to crawl a page or not. If yes then which page, etc.

You can allow or disallow bots from crawling your pages.

sher.winchan58 · 03-07-2013, 07:00 AM

Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.Robot.txt is used to prevent crawlers crawl on a certain page..

jonnysmith · 03-07-2013, 11:56 PM

The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.

champion · 03-10-2013, 10:21 PM

Its simply a text file Which is used to instruct the Search engine which files to be crawl or which not.

smithoptimizer · 03-11-2013, 12:30 AM

The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes.

geeks9 · 03-11-2013, 02:54 AM

nice dude.....thank 4 all.......

02-28-2013, 05:26 PM	#16
executiveedge01 Registered User Join Date: Feb 2013 Posts: 7	Robots.txt is only using for security purpose.It will not allow an anonymous users frequently into the website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-28-2013, 11:02 PM	#17
synchronous Registered User Join Date: Aug 2012 Posts: 32	Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-04-2013, 08:29 AM	#18
sijugk Registered User Join Date: Aug 2009 Posts: 61	Robots .txt file is the place where you can write rules for web spiders about how to crawl the site and which pages should be excluded. You can write rules specifically for each web spider. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. Last edited by sijugk; 02-24-2014 at 01:49 PM..

03-05-2013, 04:03 AM	#19
aartikrypton Registered User Join Date: Feb 2012 Posts: 32	Hi, With the help of robots.txt file search engines crawls your webpages....... Thanks Team from kryptonsoft __________________ Kryptonsoft provide you To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. has good expertise on Business Apps, Location Apps,

03-05-2013, 04:36 AM	#21
aartikrypton Registered User Join Date: Feb 2012 Posts: 32	Hello, With the help of robots.txt file search engines crawls your webpages....... Thanks Team from Kryptonsoft iPhone App Development iPhone Games Development Offshore iPhone Development __________________ Kryptonsoft provide you To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. has good expertise on Business Apps, Location Apps,

03-05-2013, 04:18 AM	#20
jesse12 Registered User Join Date: Mar 2012 Posts: 74	Robots file is a simple text file which is uploaded on root directory this is the file which is crawled by the crawler very first. So in this file we write the code which file we want to block from the crawler and other one we want to crawl. Main use of this file is just to block the admin section which generally we wish do not crawl by the search engine. Hope you understood.

03-05-2013, 04:43 AM	#22
tech.biztech Registered User Join Date: Jul 2011 Location: India Posts: 42	It is a text file which tells to search engine robots that which web page you would not like to visit. It is one kind of file or folder which tells search engine to avoid the operations described in robot.txt file. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-06-2013, 12:22 AM	#23
JACKADEN Registered User Join Date: Jun 2012 Posts: 239	Here is some information about Robot text. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. It works like this: a robot wants to visit a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds: User-agent: * Disallow: / The "User-agent: " means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. There are two important considerations when using /robots.txt: • Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention. • The /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use. So don't try to use /robots.txt to hide information. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* :: To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-06-2013, 06:25 AM	#24
spaculus Registered User Join Date: May 2012 Location: India Posts: 64	Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. , To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-07-2013, 06:31 AM	#25
thejkamath Registered User Join Date: Mar 2013 Posts: 5	Robot.txt is a file that instructs crawlers and bots whether to crawl a page or not. If yes then which page, etc. You can allow or disallow bots from crawling your pages. __________________ If you are not receiving enough comments on your blog, then read how I To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. .. For blogging tips, visit SayBlogger \| A To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-07-2013, 07:00 AM	#26
sher.winchan58 Registered User Join Date: Jul 2012 Posts: 102	Robots.txt is a file through which you can guide search engines to crawl or not to crawl certain sections of your website.Robot.txt is used to prevent crawlers crawl on a certain page.. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-07-2013, 11:56 PM	#27
jonnysmith Registered User Join Date: Mar 2013 Posts: 3	The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes. __________________ DO GOOD GET GOOD Copyright by: To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

03-10-2013, 10:21 PM	#28
champion Registered User Join Date: Feb 2013 Posts: 21	Its simply a text file Which is used to instruct the Search engine which files to be crawl or which not.

03-11-2013, 12:30 AM	#29
smithoptimizer Registered User Join Date: Mar 2013 Posts: 49	The robots exclusion standard, also known more commonly as Robots.txt, is a text file present in the root directory of a website. The Robots.txt file is a convention created to direct the activity of search engine crawlers or web spiders. The file tells the search engine crawlers which parts to web and which parts to leave alone in a website, differing between what is viewable to the public and what is viewable to the creators of the website alone. A Robots.txt file is frequently used by search engines to categorize and archive web pages, or by webmasters to proofread source codes. Last edited by smithoptimizer; 03-12-2013 at 10:36 PM..

03-11-2013, 02:54 AM	#30
geeks9 Registered User Join Date: Mar 2013 Posts: 1	nice dude.....thank 4 all....... __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
telepresence robot	hemnchy	General Discussion	1	01-02-2014 04:35 AM
Robot text.	terrijhon	Search Engine Optimization	18	06-08-2013 03:35 PM
Hidden text and links	farm gate	Google	0	11-11-2012 08:08 PM
Anchor Text Vs Google Algorithm	Jimerson12	Search Engine Optimization	0	08-16-2012 03:26 AM
Text Areas	ncjbhd	HTML / DHTML	1	11-15-2011 04:09 AM