Register on the forum now to remove ALL ads + popups + get access to tons of hidden content for members only!

Go Back   Site Owners Forums - Webmaster Forums > Search Engine Optimization > Search Engine Optimization

Notices


Reply
 
Thread Tools Rate Thread Display Modes
Old 03-12-2019, 10:05 PM   #1
Peter97
Registered User
 
Join Date: Jul 2016
Location: Romford, UK
Posts: 134
Question Is there any limit to robot.txt file

Is there any limit to robot.txt file
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
,
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
Peter97 is offline   Reply With Quote

Old 03-14-2019, 11:30 PM   #2
vinukum
Registered User
 
Join Date: Feb 2018
Location: Bangalore
Posts: 250
Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out in google. A maximum file size may be enforced per crawler.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
vinukum is offline   Reply With Quote
Old 03-15-2019, 02:37 AM   #3
peterlee1768
Registered User
 
Join Date: Sep 2015
Location: Augusta
Posts: 139
Yes, robots.txt file must be smaller than 500 KB.If you have a giant robots.txt file, remember that Googlebot will only read the first 500kb.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
peterlee1768 is offline   Reply With Quote
Old 03-15-2019, 02:37 AM   #4
peterlee1768
Registered User
 
Join Date: Sep 2015
Location: Augusta
Posts: 139
Yes, robots.txt file must be smaller than 500 KB. If you have a giant robots.txt file, remember that Googlebot will only read the first 500kb.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
|
To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
peterlee1768 is offline   Reply With Quote
Old 04-24-2019, 10:24 PM   #5
RH-Calvin
Registered User
 
Join Date: May 2015
Location: Forum
Posts: 2,118
Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.
__________________

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
| $1 VPS Hosting

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
| Windows Server with Remote Desktop Access

To view links or images in signatures your post count must be 10 or greater. You currently have 0 posts.
| Servers with Free IPMI Setup
RH-Calvin is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
What is the limit of a robot.txt file? partee Review My Website 4 08-13-2019 04:26 AM
What is the limit of a robot.txt file? NandiniShinde Search Engine Optimization 23 07-25-2018 11:49 PM
What is the limit of a robot.txt file? artijain1900 Search Engine Optimization 9 04-05-2018 03:19 AM
What is the limit of a robot.txt file? ninadordev Search Engine Optimization 3 02-16-2018 04:27 AM
What is the limit of a robot.txt file? ruhikhan Search Engine Optimization 7 01-22-2018 03:28 AM


All times are GMT -7. The time now is 04:11 PM.


Powered by vBulletin Copyright © 2013 vBulletin Solutions, Inc.