What changed at Cloudflare, and why it can make your business invisible to AI
On July 1, 2026, Cloudflare announced that starting September 15, 2026, every new domain onboarding to its network will block «Training» and «Agent» AI bots by default on any page that displays advertising. Only «Search» bots will remain allowed automatically. If your blog or corporate site runs on Cloudflare, this could mean ChatGPT, Perplexity, or Gemini stop being able to read your content to answer your customers’ questions — even while your site still ranks fine on Google.
The good news: if your site is already on Cloudflare before September 15, the default setting won’t silently change. But many B2B SMEs spin up new domains, switch providers, or simply never reviewed this setting — and could be unknowingly blocking the traffic that today comes from GEO (Generative Engine Optimization), meaning an AI model citing you as a source.
The 3 types of AI bots you need to tell apart
Cloudflare classifies AI traffic into three categories, and each plays a different role for your business:
- Search: crawl your site to cite it in AI answers (e.g. Claude-SearchBot, OAI-SearchBot). These are the ones you always want to let through — the Googlebot equivalent for the AI era.
- Agent: act when a user asks an AI assistant to research or buy something on their behalf (e.g. an agent comparing B2B vendors for a prospective buyer). Blocking these means losing «invisible» visits that actually convert.
- Training: collect content to train future models. Many companies choose to block these to protect original content, without losing visibility in search or AI answers.
The real risk isn’t blocking training bots — it’s accidentally blocking Search and Agent bots, the ones bringing you customers via ChatGPT or Gemini right now.
How to check if your site is already blocking AI bots without knowing it
Before changing anything, check your current status with these three steps:
- Open your Cloudflare dashboard → Security → Bots → AI Crawl Control (or «Bot Management» on older plans) and check which categories are allowed.
- Visit your-domain.com/robots.txt and look for
Disallowlines next to names likeGPTBot,ClaudeBot,Claude-SearchBot,PerplexityBot, orGoogle-Extended. - If you run WordPress, check whether a security plugin (firewall, anti-scraping) is blocking AI user-agents at the server level, outside Cloudflare’s control.
Step-by-step guide to configure Cloudflare without losing AI visibility
If you confirmed something is blocked, or want your setup to be ready before September, follow these steps:
- Always allow the Search category. It’s what feeds your brand’s citations in ChatGPT, Perplexity, and Google AI Overviews.
- Consider allowing Agent if your product or service is bought or quoted through comparison (software, professional services, B2B supplies): these are real users delegating research to an assistant.
- Decide on Training case by case. If your content is your competitive edge (proprietary methodologies, reports, pricing), blocking it is reasonable and doesn’t hurt your visibility in search or AI answers.
- Use the new «use» parameter of Content Signals in your robots.txt if you manage the file manually, to explicitly declare what use you allow for each bot.
- Check again in August, before the default change takes effect on September 15, especially if you’re switching providers or launching a new domain.
Not on Cloudflare? The same problem can still hit you
Other hosting and CDN providers are adopting similar default-block policies for AI bots, following the same «protect monetized content» logic. If you use a different provider, explicitly ask your technical team or agency to confirm the status of robots.txt and any web application firewall (WAF) regarding the main AI user-agents. This matters even more if you’re simultaneously investing in getting Google to feature your brand inside AI-generated answers: optimizing content is pointless if the bot can never read it.
The decision your B2B SME needs to make this week
There’s no single «block everything» or «allow everything» answer: it depends on whether your competitive edge lives in the content itself (block Training) or in being found and recommended by an AI assistant (allow Search and Agent). What’s a mistake, without exception, is not reviewing your settings and discovering in October that your AI-driven traffic dropped to zero because a default changed and nobody noticed.
If you’d like us to review your site’s setup alongside your GEO strategy, Boostify can run a quick audit before Cloudflare’s change takes effect.
