Site Owners Forums - Webmaster Forums

Site Owners Forums - Webmaster Forums (http://siteownersforums.com/index.php)
-   Search Engine Optimization (http://siteownersforums.com/forumdisplay.php?f=16)
-   -   What is the purpose of using robots.txt file in seo? (http://siteownersforums.com/showthread.php?t=179621)

nathanleo 12-31-2021 03:26 AM

A robots. txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.


All times are GMT -7. The time now is 10:15 PM.


Powered by vBulletin Copyright © 2020 vBulletin Solutions, Inc.