What is the use of Robots.txt? - Page 2 - Site Owners Forums

Digvijay Singh · 06-10-2016, 01:18 PM

Go for rustyblogger.com get full details of robot.txt there.

seomarqetrix · 06-10-2016, 09:53 PM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

jaysh4922 · 06-12-2016, 11:51 PM

Robots.txt is text file inserted into the website to contain instructions for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling. They help to control the search engine robots in your website.

webexpert · 06-13-2016, 12:31 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

ruthanitha · 06-15-2016, 01:26 AM

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

acuubek · 06-15-2016, 01:30 AM

Thanks for sharing this information.

uspromodeals · 06-15-2016, 05:50 AM

Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information.

Giraz john · 06-15-2016, 05:59 AM

robots.txt it will be use for search engines bots instruction to index and noindex the site and subfolder of the site. webmaster can stop pages and files through robots.txt

06-10-2016, 01:18 PM	#16
Digvijay Singh Registered User Join Date: Mar 2016 Posts: 39	Go for rustyblogger.com get full details of robot.txt there. __________________ Know more about To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \|\| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \|\| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \|\| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-10-2016, 09:53 PM	#17
seomarqetrix Registered User Join Date: Mar 2016 Posts: 192	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: " means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-12-2016, 11:51 PM	#18
jaysh4922 Registered User Join Date: Dec 2012 Posts: 657	Robots.txt is text file inserted into the website to contain instructions for search engine robots. The file lists webpages that are allowed and disallowed from search engine crawling. They help to control the search engine robots in your website. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-13-2016, 12:31 AM	#19
webexpert Registered User Join Date: Apr 2015 Location: Chandigarh India Posts: 23	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: " means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-15-2016, 01:30 AM	#21
acuubek Registered User Join Date: Jun 2016 Posts: 73	Thanks for sharing this information. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-15-2016, 01:26 AM	#20
ruthanitha Registered User Join Date: Jan 2016 Location: India Posts: 42	Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.

06-15-2016, 05:50 AM	#22
uspromodeals Registered User Join Date: Jun 2016 Location: california Posts: 27	Use of Robots.txt - The most common usage of Robots.txt is to ban crawlers from visiting private folders or content that gives them no additional information. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

06-15-2016, 05:59 AM	#23
Giraz john Registered User Join Date: Sep 2011 Posts: 99	robots.txt it will be use for search engines bots instruction to index and noindex the site and subfolder of the site. webmaster can stop pages and files through robots.txt __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Crawlers, spider & robots.txt?	jackthomas087	Search Engine Optimization	0	04-22-2014 03:29 AM
Evil bots consuming all my sites traffic even after i disallowed them in robots.txt	Fking	General Discussion	1	12-13-2013 11:27 PM
How to edit virtual robots.txt file in wordpress?	geniusoptimizer	PHP / mySQL	1	04-12-2013 12:36 PM
The robots.txt file	rubberfender	Yahoo	0	12-02-2012 07:32 PM
Unable to delete robots.txt file	notfake	Yahoo	0	04-29-2012 02:19 AM