Which AI bots should you let in? A decision memo

The memo, ready to send

1. Subject: our position on AI bots and agents, for your reply by [date].
2. We treat AI visitors as four kinds: training crawlers, AI search crawlers, assistants that fetch a page for a person, and agents that act for a customer.
3. Our position is: training [allow / refuse], search [allow / refuse], assistants [allow / refuse], agents acting for customers [allow / allow verified only / refuse].
4. Please export the current AI bot settings from our CDN or bot protection and tell us, for each of Search, Agent and Training traffic, whether it is allowed, challenged or blocked, and on which pages.
5. Please confirm in writing that robots.txt and the bot protection say the same thing, or list the differences.
6. Please tell us who last changed these settings and when, and name the owner we should review them with each quarter.

Where the options are

The three common positions, what each costs and the facts behind them are on the decision page. Choose the bracketed answers there. The memo only asks for confirmation; it does not settle which position is right for you.

Why four kinds

• 30 Sep 2026

  Cloudflare said on 30 September 2026 that daily requests from AI agents on its network grew by more than 1,700 percent over the past year, and that an agent comes back each time someone asks about a piece of content.

  Cloudflare: The Internet has a second audience (30 Sep 2026), read 4 Nov 2026
• 1 Jul 2026

  Cloudflare groups AI traffic into Search, Agent and Training and lets every plan, including Free, block each on all pages, on pages with ads only, or not at all (changelog of 1 July 2026).

  Cloudflare changelog: New options to manage AI traffic (1 Jul 2026), read 4 Nov 2026

Related

• Which AI bots to let inThe decision, its options and the facts.
• Ask your web teamThe sentences to send, grouped by area.