Every small publisher faces the same tension: AI bots crawl your content at scale yet send little traffic back. Block everything and you protect your work — yet you may vanish from AI answers, a growing source of discovery. Allow everything and you give your content away free.
Cloudflare AI crawl control replaces one blunt on/off switch with three bot categories, each with its own rule. This guide gives the small-site decision: what to allow, block, and charge — and the tradeoff you must not get wrong.
What Cloudflare AI Crawl Control Does
Cloudflare launched its three-category system in July 2026, replacing the single “Block AI bots” preset, deprecated September 15, 2026. If you still use the old preset, migrate now — all new controls live in the three-category system.
The system sorts AI bots into three categories based on what they do with your content:
- Search: Crawlers that index content so it can surface in AI search results, such as OAI-SearchBot.
- Agent: Agents acting in real time for a person, such as ChatGPT-User fetching a requested page.
- Training: Crawlers that collect content to train AI models, such as GPTBot and ClaudeBot.
This distinction matters: a Search crawler can bring you cited traffic, while a Training crawler takes your content and gives you nothing back. Treating them the same way is the mistake most site owners make.
The New Defaults: A Good Starting Point, Not a Strategy
From September 15, 2026, the defaults are: Search set to Allow, Agent and Training set to Block on pages with ads — applying to new customers, new sites from existing customers, and free-tier users who did not opt out before the deadline.
Those defaults suit many small sites — but they are not a strategy. “Block on pages with ads” is the key middle ground: content pages stay crawlable for traffic, while ad pages are protected, since a bot summary there replaces a paying visit.
Cloudflare AI Crawl Control Settings: Allow, Block on Ad Pages, or Block
For each category, choose Allow (all pages), Block on pages with ads (ad pages only; content pages stay open), or Block (every page).
Here is the recommended setup for a typical small site:
| Bot category | What it does | Recommended setting for a small site |
|---|---|---|
| Search (e.g. OAI-SearchBot) | Indexes content for AI search answers | Allow |
| Agent (e.g. ChatGPT-User) | Fetches pages in real time for a user | Block on pages with ads |
| Training (e.g. GPTBot, ClaudeBot) | Collects content for model training | Block on pages with ads (or Block, if you never want your work trained on) |
Allowing Search is the visibility play: if crawlers cannot read your pages, your brand cannot be cited in AI answers. Blocking Training on ad pages protects revenue without losing that visibility. Agent sits in between — a real user triggered the fetch, so some traffic is legitimate, but it deserves no free pass on monetized pages.
Warning: Blocking the Wrong Category Can Make You Invisible
Before you touch any setting: blocking the wrong category can block multi-purpose crawlers such as Googlebot and Bingbot, and if a crawler cannot read your pages, your brand cannot be cited in AI answers. Blocking everything kills AI search visibility.
Some crawlers are mixed-use, combining search, agent, and training functions, and Cloudflare judges them by all their behaviors — so one can be blocked on ad pages even if search access is permitted. Never set “Block everything” and walk away: it feels safe but costs you a growing discovery channel.
Should I Block GPTBot?
“Should I block GPTBot” is one of the most searched versions of this problem. GPTBot is a Training crawler: it collects content for model training, sends no traffic, and never cites you. For most small sites, blocking it on ad pages (or entirely) is right — you lose nothing measurable and stop donating work to a training set.
But that answer is not “block all AI bots.” GPTBot and OAI-SearchBot are different bots doing different jobs, and the search, agent, and training categories exist precisely so you can block the first without blocking the second. Block by category, not by panic.
Cloudflare Pay Per Crawl: When Charging Bots Makes Sense
On September 30, 2026, Cloudflare added opt-in controls to Pay Per Use (formerly Pay Per Crawl): publishers charge crawlers via the HTTP 402 “Payment Required” response. A crawler requesting a paid URL gets a price back — and pays, if configured to do so.
For most small sites this is “watch and wait,” not a day-one money maker: charging works only if crawlers will pay for your content, which depends on how unique it is. Original research or proprietary data gives you leverage; republished information does not, and a price tag may simply send the crawler elsewhere — fine, if it was a Training bot. Start with the category rules, watch which crawlers request your pages, then test a price on your most valuable content.
How to Set Up Cloudflare AI Crawl Control in 10 Minutes
- Log in to your Cloudflare dashboard and select your site.
- Go to Security Settings and choose Configure AI bot policies (AI Crawl Control).
- Set Search to Allow, Agent to Block on pages with ads, and Training to Block on pages with ads.
- If you still have the old “Block AI bots” preset, migrate to the three-category system now.
- Wait 24 hours, then review results under Security, then Analytics, to see which bots were allowed and blocked.
- Check ad revenue and AI referrals for two weeks before changing anything else.
On the free plan, detection uses user-agent strings — reasonable but spoofable, so verify in Analytics rather than trusting a setting alone. Use a solid technical SEO checklist to separate a bot problem from an ordinary site problem.
Also note: ChatGPT accounts for roughly three-quarters of AI referral traffic, as reported by Pulse Agency in October 2026, and leans on Bing’s index — so set up Bing Webmaster Tools alongside your Cloudflare rules.
What to Watch After You Change the Rules
Rankings shift for many reasons, so if traffic moves after a bot-policy change, check the broader picture before blaming Cloudflare — our look at Google ranking volatility in October 2026 shows how noisy one week can be. Watch four signals:
- AI referrals: Are tools still sending visitors and citing you?
- Ad revenue on protected pages: Did blocking Agent and Training change earnings?
- Blocked-bot volume in Analytics: Which bots hit most, and are legitimate ones caught?
- Search visibility: Are your pages still appearing in AI answers for target topics?
If AI referrals fall to zero, you blocked too much — loosen Search first. If ad revenue is unchanged and bot volume is huge, tighten Training to a full Block.
FAQ
Will blocking AI bots hurt my Google rankings?
Not directly — Googlebot is a search crawler, not a training bot. But blanket blocking can catch multi-purpose crawlers such as Googlebot and Bingbot. Block by category, allow Search, and verify in Analytics after 24 hours.
What is the difference between Search, Agent, and Training bots?
Search bots index content for AI results (OAI-SearchBot). Agent bots fetch pages in real time at a user’s request (ChatGPT-User). Training bots collect content for model training (GPTBot, ClaudeBot). Search can bring traffic; Training takes your work, sends none back.
Should a small site charge bots with pay-per-use?
Only once you know which bots want your content. Pay Per Use returns a price via HTTP 402, but a crawler pays only if it values your content enough. Start with the category rules, observe demand in Analytics, then test pricing on your most original content.
Is AI Crawl Control available on the free plan?
Yes — the three-category controls are on the free plan, with user-agent-based detection. The September 15, 2026 defaults (Search allowed; Agent and Training blocked on ad pages) applied to free-tier users who did not opt out, plus new customers and sites.
Conclusion
Cloudflare AI crawl control gives small site owners a precise, category-by-category choice instead of a panic button. Allow Search so AI answers can find and cite you, block Training and Agent on ad pages so scraping stops competing with revenue, and leave content pages open for discovery.
Do not block everything. Do not allow everything. Set the three rules, check Analytics after 24 hours, and adjust based on what you see — that is how you protect your work without going invisible. For help tuning the setup, reach us through our services or contact page.
Further reading: Cloudflare’s getting-started guide, a walkthrough of how to block AI crawlers, reporting on Cloudflare’s controls, and Pulse Agency’s AI search analysis behind the ChatGPT figure above.
