robots.txt - Site Owners Forums

wheelbarrow001 · 07-26-2012, 11:40 PM

The robots.txt is a stored in the root directory of the web site under ASCII coding text files, it usually against Internet search led also called web spiders, which of the site should not be content is search engine for the rovers, which can be obtained

jesse12 · 07-27-2012, 02:51 AM

Hello

Robots.txt is the file which we used to block the content from crawler so that search engine will not crawled and index our data which we mention in robots file.

Mainly the confidential data which we want to hide block with the help of robots file.

Hope you understand.

raj05 · 07-27-2012, 04:10 AM

Yes robot.txt is helpful to block crawler to crawl your website. In this your mention that area where you dont want to visit crawler.

07-26-2012, 11:40 PM	#1
wheelbarrow001 Registered User Join Date: Nov 2011 Posts: 35	robots.txt The robots.txt is a stored in the root directory of the web site under ASCII coding text files, it usually against Internet search led also called web spiders, which of the site should not be content is search engine for the rovers, which can be obtained __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

07-27-2012, 04:10 AM	#3
raj05 Registered User Join Date: May 2012 Location: india Posts: 79	Yes robot.txt is helpful to block crawler to crawl your website. In this your mention that area where you dont want to visit crawler. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Robots.txt	sameerseo	Search Engine Optimization	26	09-28-2012 04:18 AM
issue in robots.txt file	davikerkrish	Search Engine Optimization	0	07-20-2012 12:40 AM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM
What Is Robots.txt?	samlko	Search Engine Optimization	13	03-09-2012 02:37 PM
sitemap.xml and robots.txt?	jamesranatte	Search Engine Optimization	5	01-31-2012 11:16 PM

07-27-2012, 02:51 AM	#2
jesse12 Registered User Join Date: Mar 2012 Posts: 74	Hello Robots.txt is the file which we used to block the content from crawler so that search engine will not crawled and index our data which we mention in robots file. Mainly the confidential data which we want to hide block with the help of robots file. Hope you understand.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)