Cloudflare data paints a stark picture. Anthropic’s bots crawl thousands of pages for every single referral they send back to publishers. The gap dwarfs those of OpenAI, Perplexity and even Google. And the numbers come at a moment when bots have already overtaken human traffic on the internet.
Business Insider reported the latest figures Tuesday. Anthropic’s ClaudeBot registered a 2,800-to-1 crawl-to-referral ratio for the first week of July. That mark improved from earlier peaks. Yet it still towered over OpenAI’s lower multiple, Perplexity’s smaller figure and Google’s modest 5-to-1 benchmark. DuckDuckGo sat closest to balance at roughly 3-to-1. (Business Insider)
These disparities stretch back further. Cloudflare’s own analysis from last summer showed Anthropic hitting 38,000 crawls per referral in July 2025. The ratio had dropped from an eye-watering 286,000-to-1 in January that year. Training runs accounted for nearly 80 percent of AI crawling activity. Search and user-driven requests trailed far behind. (Cloudflare)
But the pattern holds. AI companies vacuum up vast swaths of public web content. They return little traffic in exchange. Publishers watch their Google referrals decline as AI overviews and chat interfaces answer questions directly. The old social contract — crawl my pages, send me visitors — frays.
Cloudflare draws a line
The infrastructure giant didn’t stop at publishing data. On July 1 it announced fresh defaults. Starting Sept. 15, training and agent crawlers will be blocked by default on any page that displays ads. Search crawlers stay allowed. Mixed-use bots that blend purposes face the stricter rule. New customers and free-tier sites receive the updated policy first. (Cloudflare)
Cloudflare CEO Matthew Prince framed the move plainly. “AI was taking everything and sending back nothing, presenting an existential threat to website owners.” He added that with non-human traffic now the majority, the industry must “act faster so that a sustainable ecosystem can emerge.” TechCrunch covered the policy shift the same day. It noted partnerships with Ceramic.ai and You.com to test pay-per-use models where AI firms compensate publishers when content generates value. (TechCrunch)
More than half of AI crawler requests already hit unchanged pages, Cloudflare data shows. Blocking training bots could ease server loads. Publishers gain levers to protect ad inventory while preserving discoverability through legitimate search. Small sites especially need that visibility. Yet the change forces AI operators to separate their bots or negotiate deals. No more one-size-fits-all scraping.
Anthropic itself sits in an awkward spot. The company has loudly condemned “distillation attacks.” These campaigns use thousands of fake accounts to query its Claude models repeatedly. Attackers extract reasoning patterns, tool-use behaviors and chain-of-thought data. They then train rival systems on the outputs. The method strips away safety guardrails. It accelerates foreign labs toward frontier capabilities without original research costs. (Anthropic)
In June Anthropic accused Alibaba’s Qwen team of running the largest known such operation. Nearly 25,000 fraudulent accounts generated almost 29 million queries, according to letters obtained by CNBC and Reuters. The campaign targeted advanced features. Anthropic warned it could speed China’s progress toward systems like its Mythos model. National security implications loomed large. Distilled models lack safeguards. Authoritarian governments could deploy them for cyber operations or worse. (CNBC) (Reuters)
Yet critics see parallels. Anthropic’s own bots consume web content at massive scale. They distill patterns from news articles, blog posts and forums into model weights. Publishers receive scant referral traffic. The company updated its crawler documentation to list separate bots for training, search indexing and user requests. Still, verification gaps remain. Some AI crawlers ignore robots.txt or spoof identities.
The hypocrisy charge lands. Anthropic positions itself as the responsible actor in AI safety. It bars access from China and pushes export controls on advanced hardware. Distillation, it argues, lets adversaries bypass those controls by copying capabilities through queries rather than chips. But the web provides the raw material. And that material flows one way.
Recent weeks brought more friction. Cloudflare’s July 2026 rankings triggered fresh discussion on X. Users noted Anthropic’s ratios, even as they improved, still dwarfed competitors. One post highlighted the drop from earlier peaks yet questioned whether search additions truly balanced the ledger. Another tied the numbers to broader US-China tensions. Alibaba responded to Anthropic’s accusations by banning employee use of Claude Code over claimed backdoor risks. The feature was an experimental anti-abuse tool designed to detect distillation attempts. China called it a privacy threat.
These episodes reveal deeper tension. AI progress depends on data. Public web data remains the cheapest, richest source. Yet the economics no longer favor creators. Referrals from traditional search have fallen. AI interfaces synthesize answers without clicks. Training runs repeat crawls on static content. One Cloudflare chart showed training’s share of AI bot activity climbing from 72 percent in 2024 to 79 percent last summer. Spikes hit during spring 2025. Growth then slowed. The appetite, however, endures.
Publishers face hard choices. Block everything and lose search visibility. Allow training bots and subsidize competitors. Or demand payment. Cloudflare now offers tools for the last path. Its BotBase and content-use signals in robots.txt give finer control. Signals can specify immediate, reference or full use. The industry inches toward explicit contracts.
Not every actor behaves alike. ByteDance’s crawler once dominated shares but fell sharply. Its referral ratio flipped below 1-to-1. Google maintains a low multiple even as its AI Overviews reduce clicks. OpenAI improved its balance modestly. Anthropic added search indexing in 2025. Its July 2026 ratio reflects that effort. Progress exists. The absolute volumes still tilt heavily toward extraction.
So what comes next? More defaults like Cloudflare’s. More lawsuits over fair use. More experiments with licensed data pools and micropayments. Some AI labs already strike direct deals with publishers. Others train on synthetic data to reduce web dependence. Yet frontier models still crave fresh, human-generated text.
The web’s original bargain assumed mutual benefit. Crawlers indexed. Humans visited. Ads paid the bills. That loop weakens when answers appear in chat windows and training servers hum 24 hours a day. Anthropic’s numbers throw the imbalance into sharp relief. They also underscore the company’s own complicated stance. It decries model theft while its bots harvest content at scale.
Industry insiders watch closely. The September deadline approaches. AI companies must decide whether to split their crawlers, pay for access or accept blocks on ad-supported sites. Publishers will tune settings case by case. Some will prioritize revenue. Others may value visibility. The data Cloudflare publishes each month will keep score.
Balance remains elusive. But the conversation has shifted. No longer abstract ethics. Concrete ratios. Concrete policy changes. Concrete dollars. The web’s content creators finally hold cards. How they play them will shape what data tomorrow’s models learn — and who profits from it.
WebProNews is an iEntry Publication