Cloudflare will block mixed-use web crawlers from accessing ad-supported sites starting September 15, 2026. This policy aims to give website owners more control over how AI companies use their content, potentially affecting AI models' access to web data.
Cloudflare announced that starting September 15, 2026, it will block mixed-use web crawlers that index websites for both traditional search engines and AI model training, especially on ad-supported pages. This action comes as part of its initiative to give more control to website owners regarding how their content is used by AI companies.
The decision is anticipated to impact AI companies by restricting their access to web resources traditionally used for training and enhancing AI models and services. Many website owners want their content accessible via search but seek protection against their intellectual property being exploited without compensation.
Previously, Cloudflare offered these blocking functions as optional settings. However, with increasing non-human web traffic, the company has decided to make these restrictions default for new customers, newly set up sites, and existing free users. This measure is intended to encourage more transparent AI development practices.
By instituting this policy, Cloudflare aims to shift the dynamics of traffic patterns on the internet. Matthew Prince, CEO of Cloudflare, remarked on the need for a sustainable ecosystem that balances technological advances with content ownership rights. The move also encourages AI companies to separate the functions of search and agent-based data gathering.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
Cloudflare will start automatically blocking web crawlers that index for AI and search functions by default. This change, effective September 15, 2026, aims to give website owners more control over AI usage of their content and restore fairness as AI traffic patterns shift.
Cloudflare announced it will block mixed-use web crawlers from accessing ad-supported sites beginning September 15, 2026. This policy aims to protect publishers' content, impacting AI companies' access to web resources for training and services.