New Data Confirms Googlebot’s 2 MB Crawl Limit Is Adequate

Most Web Pages Are Far Below the Limit

New data extracted from the HTTP Archive shows that Googlebot’s 2-megabyte (MB) HTML crawl limit is not a worry for most websites. In fact, a good majority of pages fall well below this threshold.

Googlebot downloads only the raw HTML of a page during crawling. Here, the markup, document structure, inline scripts, and inline styles are considered without taking into account external CSS or JavaScript files.

According to the latest HTTP Archive data, the median HTML size of web pages is just 33 kilobytes (KB). Even at the 90th percentile, HTML size reaches only 155 KB. That is still much lower than Google’s 2 MB crawl cap.

In a nutshell, 2 MB equals over two million characters of text. Very few web pages contain that much raw HTML.

What the HTTP Archive Data Shows?

The HTTP Archive explains that “HTML bytes” refers to the textual weight of the markup on a page. This includes elements such as <div> and <span>, along with inline scripts and styles that can increase page size.

However, the data shows that HTML size remains relatively small across the web:

• Median HTML size is 33 KB
• 90th percentile is 155 KB
• Only extreme outliers approach or exceed 2 MB

Significant size increases appear only at the 100th percentile. At that level, desktop HTML can reach 401.6 MB and mobile HTML 389.2 MB and are clearly rare and abnormal cases.

This confirms that pages exceeding 2 MB are extreme exceptions, not normal.

Home Pages vs. Inner Pages

The data also compares home pages with inner pages. Surprisingly, there is little difference in HTML weight between them.

Differences become noticeable only above the 75th percentile. At the extreme 100th percentile, inner pages can grow dramatically larger. Such as:

• Inner page HTML: 624.4 MB
• Home page HTML: 166.5 MB

Moving ahead, these figures indicate rare trends. For a good number of websites, both home pages and internal pages remain well below the 2 MB threshold.

Mobile and Desktop Sizes are in Effect Similar

Another interesting news from the HTTP Archive is that mobile and desktop HTML sizes are almost the same.

This suggests most websites serve a single unified version of their HTML to both device types. While this can increase total page weight slightly, it simplifies development and maintenance.

More importantly, even with combined code for multiple device types, HTML sizes still remain far under Googlebot’s 2 MB crawl limit for nearly all sites.

Tools to Check Your Page Size

Although this issue affects very few websites, there are tools available for reassurance.

Dave Smart of Tame the Bots recently updated a tool that simulates Googlebot stopping at the 2 MB limit. It allows users to see how a page would appear if crawling were cut off at that point.

Other tools, such as Toolsaday’s Web Page Size Checker and Small SEO Tools’ Website Page Size Checker, help measure total page weight quickly.

These tools confirm what the data already shows: most pages are nowhere near the limit.