Site Owners Forums - Webmaster Forums - What is the limit of a robot.txt file?

Site Owners Forums - Webmaster Forums (http://siteownersforums.com/index.php)

- Search Engine Optimization (http://siteownersforums.com/forumdisplay.php?f=16)

- - What is the limit of a robot.txt file? (http://siteownersforums.com/showthread.php?t=209325)

ninadordev

02-15-2018 11:27 PM

What is the limit of a robot.txt file?

Hello friends,

What is the limit of a robot.txt file?

lilianalilly

02-15-2018 11:39 PM

there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. > A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit .

Carlbrewster

02-15-2018 11:56 PM

Despite of implementing Robots.txt, there are chances that the blocked page is crawled and indexed by Google.

Yashmin

02-16-2018 05:27 AM

Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. > A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored.

nathanleo

12-24-2021 12:29 AM

Google currently enforces a robots. txt file size limit of 500 kibibytes (KiB). Content which is after the maximum file size is ignored. You can reduce the size of the robots.

All times are GMT -7. The time now is 09:24 AM.