Cloudflare just marked the second Content Independence Day with a set of tools that give every website owner, even on the free plan, fine-grained say over how AI systems touch their content. The changes arrive one year after the company first offered a one-click block on AI bots. Back then the worry centered on training data scraped without compensation. Now the picture has grown more complex. Publishers still seek payment for original work. Yet many also want their sites to surface in search results. The old binary choice no longer fits.
The company outlined its thinking in a detailed post on its blog. “We could debate the cutoff for what qualifies as ‘AI’ today, just to find that the standard changes tomorrow,” Cloudflare wrote. So it moved past simple labels. Instead it asks three questions about any automated visitor. What is it doing on the site? What does it store? How will it reshare the material?
From those questions Cloudflare built a pragmatic taxonomy built around three core behaviors. Search crawlers index content so it can appear in results later and ideally drive referral traffic. Agent bots act in real time on a person’s behalf, often to complete a task while a human waits. Training systems ingest data permanently to improve models. The same crawler can fall into more than one category. Cloudflare now tracks all of them. It strongly encourages bot operators to split multi-purpose crawlers into separate user agents for transparency.
These distinctions matter. A small publisher might welcome Googlebot when it indexes for search yet recoil when the same bot feeds training data. Larger sites with ad inventory face different pressures. Human attention fuels their business. Bots that bypass that attention cut revenue. Cloudflare’s response is practical. On September 15, 2026, new defaults kick in. For fresh domains, Training and Agent bots will be blocked by default on pages that display ads. Search stays allowed. Multi-purpose bots that combine search indexing with training will follow the strictest rule applied to any of their behaviors. Existing customers can opt out of the default shift through their security settings.
The update builds directly on last year’s Block AI Bots feature. That tool targeted single-purpose training crawlers. The new controls expand the palette. Website owners select from presets for Search, Agent, or Training. Each can be set to block on all pages, block only on ad pages, or allow. Free-tier users gain access. The legacy Block AI Bots toggle will eventually deprecate in favor of these more precise switches. Documentation at Cloudflare’s developer site walks through the configuration steps in the dashboard.
Visibility receives equal attention. Enterprise Bot Management customers now see BotBase, a searchable database of every tracked bot and agent. The interface shows each bot’s classifications under the new taxonomy. Users can filter traffic from a specific bot, copy its detection ID, and drop it into security rules. Cloudflare describes the launch as visibility first. Controls will follow later this year. The directory already lists behaviors that range beyond the three main categories. Transact covers checkout actions. Data Collection includes price scraping. Security Testing, SEO, Ads Verification, Social Link Preview, Feed Fetching, and Monitoring round out the list.
But classification alone doesn’t tell the full story. What happens to the content after the crawler leaves? Cloudflare is testing a new signal that answers that question. It builds on the Content Signals standard and lives inside robots.txt. The signal offers three levels. Immediate means interact but store or reuse nothing. Reference, the default, permits indexing, excerpts, and links back to the source. Full allows summarization and reproduction. Website owners can combine the content-use preference with bot categories. A rule might read: allow Search, SEO, and Ads Verification bots but only up to the reference level.
Managed robots.txt files already used by many customers will automatically append the new use=reference parameter. An example in the blog shows the before and after. The older version reads “Content-Signal: search=yes,ai-train=no”. The updated line adds “,use=reference”. These entries express preference rather than enforce hard blocks. Still, bots that ignore them and reproduce content in full risk losing Verified status. Verified bots must represent themselves honestly and avoid abuse. The definition of Verified itself has shifted. It no longer equals default allowed. Instead it means the bot can pass if its categories align with the site’s chosen permissions.
The company is also opening the verification process and building tools for bot operators. A forthcoming platform will let operators confirm how Cloudflare classifies their crawlers. The goal is mutual understanding. Cloudflare wants to reduce evasion tactics that small sites already encounter. When incumbents use the same bot for search and training, they gain an edge. New entrants then hide their intentions to close the gap. Both behaviors disadvantage independent publishers.
These traffic controls sit alongside Cloudflare’s aggressive push into AI inference itself. In April the company turned its network into a unified inference layer. Developers can now call models from more than a dozen providers through a single API. The same AI.run() binding that powers Workers AI also reaches third-party offerings from OpenAI, Anthropic, Google, and others. A Cloudflare blog post by Ming Lu and Michelle Chen spelled out the vision. “One catalog, one unified endpoint.” At launch the catalog held more than 70 models across 12 providers. Image, video, and speech capabilities arrived to support multimodal work.
Latency sits at the center of the design. Agents need fast time to first token. Cloudflare’s global network, with GPUs in hundreds of cities, routes requests to the nearest capacity. The company claims its custom inference engine, written in Rust and called Infire, delivers better performance than standard serving stacks. Automatic failover keeps long-running agent sessions alive even if a provider drops. Streaming responses buffer and reconnect without double billing. Replicate has joined the effort, bringing its Cog technology so customers can eventually host their own models on the platform.
Pricing follows a serverless model. Customers pay per inference with no idle charges. A separate neuron-based system for some models has been retired in favor of simpler metering. The dashboard now centralizes spend across all providers with custom metadata tags that break costs down by team, user, or workflow. That single pane of glass matters when agents chain dozens of model calls in a single session.
Recent coverage shows the announcements landed at a moment of high tension. A Hacker News thread linked to the blog post and drew comments on the balance between openness and control. One X user captured the sentiment: publishers can finally set terms instead of playing endless robots.txt whack-a-mole. Another noted the taxonomy gives site operators a real negotiation layer with model makers. Japanese and other language posts echoed the same theme. The timing feels deliberate. Search engines increasingly answer questions directly rather than send clicks. AI agents browse on users’ behalf. Training continues in the background. Without clear signals, the open web risks becoming a one-way feed.
Cloudflare’s approach doesn’t pick winners. It gives owners the dials. Block training on ad pages while keeping search open. Permit agents only if they promise not to store data. Demand reference-only use for any bot that summarizes. The taxonomy scales as new behaviors appear. BotBase grows with the directory. Content Signals gain teeth through Verified status and future enforcement.
Critics may argue defaults still favor search giants. Googlebot, Applebot, and Bingbot carry multiple classifications. Under the new rules they could face blocks from sites that reject training. Yet the opt-out path exists. Customer choice remains the stated priority. And the company continues to notify users before defaults change.
The bigger shift may be cultural. For years the deal was simple. Crawl my pages, send me traffic. AI broke that bargain. Now Cloudflare offers a more honest one. Tell me your intent. Respect my limits. Compensate when value flows only one direction. Whether bot operators split their crawlers, whether search companies adjust, whether compensation markets emerge, all remain open questions. But the tools are in publishers’ hands. For the first time they can answer the three questions on their own terms. What are you doing? What will you keep? And what will you give back?
WebProNews is an iEntry Publication