Which AI bots and agents to let in

The decision

Decide, as four separate choices, whether you welcome training crawlers, AI search crawlers, assistants that fetch a page for a person, and agents that act for a customer, then make robots.txt and bot protection say the same thing.

Who decides: Leadership decides the position; the web or security team enforces it and reports back.

The options

1. Allow all four kindsYour content can train models, appear in AI answers and be read live for customers; you give up control over training use. Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026), read 4 Nov 2026.
2. Refuse training, allow search, assistants and agentsThe common middle path: 17 percent of Cloudflare sites block training while fewer than 1 percent block search. Cloudflare: The Internet has a second audience (30 Sep 2026), read 4 Nov 2026.
3. Block assistants and agents tooOra's lab found agents still answer when blocked, from other sources, and those answers miss more facts. Ora research: AX is the new AEO (8 Oct 2026), read 4 Nov 2026.

What we found

• 4 Nov 2026

  In the network's own crawl, run of 4 Nov 2026 (13,071 of 13,071 domains from Tranco list 746WX plus Ora leaderboard domains), 13.9 percent of 8,624 sites with a readable robots.txt disallowed GPTBot at the root, against 6.9 percent for ChatGPT-User.

  Scorecard methodology: the crawl of 4 Nov 2026, read 4 Nov 2026
• 1 Jul 2026

  Cloudflare groups AI traffic into Search, Agent and Training and lets every plan, including Free, block each on all pages, on pages with ads only, or not at all (changelog of 1 July 2026).

  Cloudflare changelog: New options to manage AI traffic (1 Jul 2026), read 4 Nov 2026
• 1 Sep 2022

  RFC 9309 states that robots.txt rules are not a form of access authorization; enforcement needs bot management.

  RFC 9309 Robots Exclusion Protocol, read 4 Nov 2026

How to decide

• Start from the customer. If an agent acting for a buyer cannot reach your prices, that buyer's task goes to a competitor.
• Refusing training while allowing the other three is the middle path in the options below. Decide it separately from search.
• Ask who last looked at the CDN setting. A rule switched on for security can undo a decision made in marketing.
• Write the position on one page with a review date, then ask the web team to confirm in writing that robots.txt and bot protection match it.

Where this shows up

• Self-check question 1Has someone in your business deliberately decided which AI crawlers and assistants your robots.txt file allows, rather than leaving the default?
• Self-check question 2Has anyone checked in the last six months whether your CDN, firewall or bot protection blocks AI crawlers or AI assistants?
• The publishing guideSeparate controls for training, search and live fetches, and what Cloudflare changed in 2026.
• The software guideAgents that read docs and pricing, sign up for trials and open paid accounts.
• The local business guideGoogle calling stores, searching reservation platforms and pointing customers to the provider.