What is the importance of robot.txt in SEO? - Site Owners Forums

vashilove · 02-04-2019, 05:30 AM

Hello,
What is the importance of robot.txt in SEO?

RH-Calvin · 02-04-2019, 11:45 PM

Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling.

albertc99 · 02-06-2019, 10:23 PM

The robots.txt file is a text file used in SEO . It is intended for search engine indexing robots to specify which pages can or can not be indexed. So any search engine starts exploring a website by looking for the robots.txt file at the root of the site.

Despite this, the importance of the robots.txt file is sometimes underestimated. Here, however, is important information on this subject. A Google employee has just indicated that if your site has a robots.txt file , and that it is not readable by Google (so if the returned http code is neither 200 nor 404), then the robot will simply stop crawling the site.

Do not forget to check if your file returns code 200. Note that if you do not have one, it will not pose any specific problem (code 404). Your site will be fully readable by all the different robots of different engines.

adibitar69 · 02-07-2019, 12:17 AM

Robots.txt is a file in website which gives instruction to search engine crawler which page of website should be indexed and which Web pages should be ignored.

rajkiran · 02-07-2019, 01:58 AM

Robots.txt is a text file located in the site's root directory for search engines Spiders and Crawlers what website pages and files you want or don't want them to visit.

Zora2012 · 02-07-2019, 05:46 AM

Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all.

vinukum · 02-12-2019, 01:08 AM

Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.
Basic format of the robot.txt file.
User-agent: [user-agent name]
Disallow: [URL string not to be crawled]

steveamith · 02-12-2019, 03:04 AM

IF you want to show your page to the user but do not want to google crawl it then we use Robot.txt.

02-04-2019, 05:30 AM	#1
vashilove Registered User Join Date: Apr 2018 Posts: 147	What is the importance of robot.txt in SEO? Hello, What is the importance of robot.txt in SEO? __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-04-2019, 11:45 PM	#2
RH-Calvin Registered User Join Date: May 2015 Location: Forum Posts: 2,517	Robots.txt is a text file that lists webpages which contain instructions for search engines robots. The file lists webpages that are allowed and disallowed from search engine crawling. __________________ █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| $1 VPS Hosting █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Windows Server with Remote Desktop Access █ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| Servers with Free IPMI Setup

02-06-2019, 10:23 PM	#3
albertc99 Registered User Join Date: Jan 2016 Posts: 49	The robots.txt file is a text file used in SEO . It is intended for search engine indexing robots to specify which pages can or can not be indexed. So any search engine starts exploring a website by looking for the robots.txt file at the root of the site. Despite this, the importance of the robots.txt file is sometimes underestimated. Here, however, is important information on this subject. A Google employee has just indicated that if your site has a robots.txt file , and that it is not readable by Google (so if the returned http code is neither 200 nor 404), then the robot will simply stop crawling the site. Do not forget to check if your file returns code 200. Note that if you do not have one, it will not pose any specific problem (code 404). Your site will be fully readable by all the different robots of different engines. __________________ *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.* , *To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.*

02-07-2019, 12:17 AM	#4
adibitar69 Registered User Join Date: Nov 2016 Location: Dubai Posts: 25	Robots.txt is a file in website which gives instruction to search engine crawler which page of website should be indexed and which Web pages should be ignored. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-07-2019, 05:46 AM	#6
Zora2012 Registered User Join Date: Nov 2012 Location: India Posts: 276	Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - Shopping Cart- PrestaShop, Zencart; CMS - WordPress, Joomla; Custom Development To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. - SEO, Pay Per Click (PPC), SEM, Link Popularity Building, SEO Copywriting, SMO, SMM

02-07-2019, 01:58 AM	#5
rajkiran Registered User Join Date: Aug 2016 Posts: 961	Robots.txt is a text file located in the site's root directory for search engines Spiders and Crawlers what website pages and files you want or don't want them to visit.

02-12-2019, 01:08 AM	#7
vinukum Registered User Join Date: Feb 2018 Location: Bangalore Posts: 283	Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. Basic format of the robot.txt file. User-agent: [user-agent name] Disallow: [URL string not to be crawled] __________________ To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts. \| To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.

02-12-2019, 03:04 AM	#8
steveamith Registered User Join Date: Feb 2018 Location: London Posts: 4	IF you want to show your page to the user but do not want to google crawl it then we use Robot.txt.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
robot.txt	Shivam Tripathi	Web Promotion	5	12-16-2018 10:11 PM
What is Robot.txt?	kumar1_16	General Discussion	11	03-29-2017 03:15 AM
how to keyword importance in seo?	ruhanikakhan	Search Engine Optimization	19	06-22-2016 09:14 AM
Importance of sub domain.	Adhitimishra23	Traffic Corner	1	12-24-2015 09:12 PM
Does Anyone Know About The Importance Of SEO?	brettlee585	Search Engine Optimization	5	11-26-2012 05:15 AM