Cloudflare AI Bot Rules Take Effect to Block Training Crawlers
Cloudflare began enforcing new default controls on September 15, 2026, automatically blocking artificial intelligence training and agent bots on ad-supported pages for new domains and unconfigured free-tier customers. Publishers, creators, and AI companies now face a shifting technical environment as web traffic is divided into distinct operational categories.
Dual-Purpose Crawlers Face Strict Limits Under New Defaults
The infrastructure firm splits automated web traffic into three primary categories: search, training, and agent. Search crawlers retain default access to index content, allowing users to discover pages through search engines. Conversely, automated systems collecting data to fine-tune AI models or acting as real-time chat-fetching bots face automatic restrictions on pages displaying advertisements.
Cloudflare first outlined the policy in July before establishing the September 15 deadline. Website owners retain the ability to allow, block, or selectively restrict each category via their dashboard.
Automated crawlers performing multiple functions, such as Googlebot, Applebot, and BingBot, trigger the most restrictive applicable setting if a site chooses to prohibit training. A crawler used for both search and model training can be entirely blocked if a publisher restricts training activity.
The enforcement arrives as publishers scrutinize the balance between content consumption by AI systems and the referral traffic those services return. Traditional search engines index pages and drive visits that generate advertising and subscription revenue, whereas generative AI services often extract information to present direct answers without sending users to the source.
To address this imbalance, Cloudflare provides analytics tools allowing select customers to track how frequently individual AI operators crawl a site compared with the referral traffic they produce. The company stated the objective is helping website owners distinguish between automated services that drive discovery and those that consume content without compensation.
