Cloudflare Report: Googlebot Dominates AI Crawler Traffic

A Cloudflare report reveals that Googlebot dominates AI crawler traffic, highlighting its significant role in indexing and web monitoring.

Cloudflare’s 2025 Radar Year in Review shows just how outsized Googlebot’s footprint is compared to other AI and search crawlers, both for traditional indexing and AI model training. While overall Internet traffic grew 19% year over year, a significant share of HTML requests now comes from AI-related bots, with Googlebot alone outpacing the rest of the AI field combined.​

Googlebot Tops AI Crawlers By Reach

Cloudflare analyzed successful HTML requests from major AI and search crawlers across October and November 2025 and found that Googlebot reached about 11.6% of unique pages in the sample.

That’s more than three times the coverage of OpenAI’s GPTBot (3.6%) and almost 200 times the reach of PerplexityBot, which saw only 0.06% of pages.​

Bingbot ranked third at 2.6% of pages crawled, followed by Meta-ExternalAgent and ClaudeBot at roughly 2.4% each. Cloudflare notes that because Googlebot is used both for search indexing and AI training, website operators effectively cannot block its AI use without risking their search visibility:​

Cloudflare wrote:

“Because Googlebot is used to crawl content for both search indexing and AI model training, and because of Google’s long-established dominance in search, Web site operators are essentially unable to block Googlebot’s AI training without risking search discoverability.”

AI Bots’ Share Of HTML Requests

Across 2025, AI bots other than Googlebot accounted for an average of 4.2% of HTML requests on Cloudflare’s network, with the share dipping as low as 2.4% in early April and climbing to 6.4% in late June.

Googlebot by itself generated 4.5% of HTML requests, slightly more than all other AI-oriented crawlers combined, and its share briefly peaked around 11% of HTML traffic in late April as crawl volume ramped up.​

Human-generated HTML traffic started the year about seven percentage points below non-AI bot traffic, but the gap narrowed over time. By early December, humans accounted for roughly 47% of HTML requests while non-AI bots represented about 44%, reflecting both growing automation and continued heavy crawler activity.​

Crawl-To-Refer Ratios: Who Crawls, Who Sends Traffic

Cloudflare’s crawl-to-refer ratio metric compares how often a bot crawls pages to how often it actually sends users back as referrals, highlighting how “extractive” different platforms are.

Among AI platforms, Anthropic showed the highest ratios in the second half of 2025 between about 25,000:1 and 100,000:1 indicating very heavy crawling relative to the traffic it returns.​

OpenAI’s ratios reached up to roughly 3,700:1 in March, while Perplexity maintained the lowest ratios among leading AI players, generally below 400:1 and under 200:1 from September onward.

For comparison, Google’s search traffic looked far more balanced, with a crawl-to-refer ratio typically between 3:1 and 30:1 over the year, reinforcing the idea that classic search still sends substantially more traffic back to publishers per crawl than current AI agents do.​

User-Action Crawling Surges With Chatbot Usage

Cloudflare distinguishes between background/model-training crawls and “user action” crawls triggered when people ask questions in chatbots that then fetch live web content.

User-action crawling grew the fastest in 2025, with volume increasing more than 15x between January and early December, closely tracking usage patterns of agents like OpenAI’s ChatGPT-User bot.​

This traffic shows a clear weekly rhythm from mid-February onward, consistent with heavier use in schools and workplaces and a noticeable dip during the June–August holiday period. It’s a useful reminder that a growing share of AI crawling is now tied directly to end-user queries rather than offline model training alone.​

How Sites Are Blocking AI Crawlers

Cloudflare’s review of robots.txt rules for nearly 3,900 of the top 10,000 domains found that AI-focused crawlers are now the most frequently blocked user agents. GPTBot, ClaudeBot, and CCBot had the largest number of full Disallow directives, meaning many sites have chosen to block them from accessing any content at all.​

Googlebot and Bingbot were treated differently. Their disallow rules skewed heavily toward partial blocks (for example, login paths or internal tools) rather than full-site bans, reflecting the ongoing dependence on traditional search traffic even as publishers push back against AI-only scrapers.​

Civil Society Now The Top Attack Target

Outside of crawling, Cloudflare’s security data shows a notable shift in which sectors attract the most attacks. For the first time, organizations in the “People and Society” category religious groups, nonprofits, civic organizations, libraries became the most targeted vertical, receiving 4.4% of global mitigated traffic over the year.​

Attack volume against this group spiked above 17% of mitigated traffic in late March and peaked at 23.2% in early July, with many of the affected organizations shielded under Cloudflare’s Project Galileo protection program.

Gambling and gaming, the top target in 2024, saw its share of attacks fall by more than half to 2.6% in 2025.​

Other Notable Internet Trends

Cloudflare’s report highlights several broader infrastructure and traffic shifts:

• Global Internet traffic rose 19% year-over-year, with growth concentrated in the second half of 2025.​
• Post-quantum encryption adoption nearly doubled, securing 52% of human traffic to Cloudflare by year-end, up from 29% at the start of the year.​
• ChatGPT remained the leading generative AI service worldwide, while Google Gemini, Windsurf AI, Grok/xAI and DeepSeek emerged as new top-10 players.​
• Starlink traffic doubled as the service expanded into more than 20 additional countries.​
• Of 174 major Internet outages observed, almost half were attributed to government-directed shutdowns; cable-cut outages fell by about 50%, while power-related outages doubled.​
• Europe dominated network quality metrics, with Spain ranked first for overall Internet quality and average download speeds above 300 Mbps among measured countries.​

Why This Matters For Site Owners And SEOs

For publishers, the data quantifies a growing tension: AI platforms are crawling aggressively but, with a few exceptions, sending relatively little traffic back, especially compared to traditional search.

Google’s dual-purpose Googlebot sits in a unique position, making it practically impossible to opt out of AI training without also sacrificing core search visibility.​

The crawl-to-refer numbers and robots.txt patterns provide a baseline for how the ecosystem looks heading into 2026.

Bottom Line

If AI assistants evolve into more explicit search products with better attribution and referrals, those ratios may improve but right now, most AI crawlers remain net consumers of bandwidth and content with limited return traffic for site owners.