Why We Use robot.txt file In Seo..? - Site Owners Forums

ankitasharma123 · 04-19-2017, 10:49 PM

Hello friends,

Why We Use robot.txt file In Seo..?

orionxander · 04-19-2017, 10:55 PM

When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

johnkevin · 04-19-2017, 11:31 PM

The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website.

Hashmicro · 04-19-2017, 11:34 PM

Robots.txt is a text file you put on your site to tell search robots which pages you wouls like them not to visit

praveenitech30 · 04-20-2017, 04:18 AM

Robots.txt is a text file you put on your site to tell search robots which pages you wouls like them not to visit

cultratradseo · 04-20-2017, 08:59 AM

In a nutshell. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

tourism123 · 04-20-2017, 10:11 AM

Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

bricksmith · 04-20-2017, 12:59 PM

You can use Google Search Consol to test your sites robots.txt. I definitely recommend doing this to make sure it is working properly

04-19-2017, 10:49 PM	#1
ankitasharma123 Registered User Join Date: Aug 2013 Posts: 332	Why We Use robot.txt file In Seo..? Hello friends, Why We Use robot.txt file In Seo..? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-19-2017, 10:55 PM	#2
orionxander Registered User Join Date: Mar 2017 Posts: 224	When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-19-2017, 11:31 PM	#3
johnkevin Registered User Join Date: Oct 2016 Posts: 259	The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

04-19-2017, 11:34 PM	#4
Hashmicro Registered User Join Date: Nov 2016 Posts: 515	Robots.txt is a text file you put on your site to tell search robots which pages you wouls like them not to visit __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* \| *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

04-20-2017, 04:18 AM	#5
praveenitech30 Registered User Join Date: Jan 2017 Posts: 108	Robots.txt is a text file you put on your site to tell search robots which pages you wouls like them not to visit __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-20-2017, 08:59 AM	#6
cultratradseo Registered User Join Date: Apr 2017 Posts: 74	In a nutshell. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol. The "User-agent: " means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

04-20-2017, 10:11 AM	#7
tourism123 Registered User Join Date: Mar 2017 Posts: 264	Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

04-20-2017, 12:59 PM	#8
bricksmith Registered User Join Date: Apr 2017 Location: Arizona, USA Posts: 16	You can use Google Search Consol to test your sites robots.txt. I definitely recommend doing this to make sure it is working properly __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. // To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. // To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. // To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
What is robot.txt file and what are the benefits of using it..?	AshokDixit89	Search Engine Optimization	8	05-05-2017 04:30 AM
What is robot.txt file and what are the benefits of using it?	Martinricky	Social Networks	31	12-21-2016 02:46 AM
Which pages should be put in robot.txt file?	Martinricky	Search Engine Optimization	1	07-24-2016 11:15 PM
how can create the Robot.txt file	eagleeye dental	Search Engine Optimization	0	09-01-2014 09:40 PM