What is the limit of a robot.txt file?
What is the limit of a robot.txt file?
|
In this section of our guide to robots directives, we’ll go into more detail about the robots.txt text file and how it can be used to instruct the search engine web crawlers. This file is especially useful for managing crawl budget and making sure search engines are spending their time on your site efficiently and crawling only the important pages.
|
A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.
|
Quote:
|
Well it didn't know that thanks for sharing this info!
|
One XML file can include maximum 50000 urls.
|
50000 i guess
|
All times are GMT -7. The time now is 07:36 PM. |
Powered by vBulletin Copyright © 2020 vBulletin Solutions, Inc.