--- title: "Which AI bots and agents to let in | Am I Ready for Agents?" description: "Which AI bots and agents to let in: the decision, three options with what each one costs, dated facts and the sentence to send your web team." canonical: https://amireadyforagents.com/decisions/which-bots-to-allow site: "Am I Ready for Agents?" last_updated: 2026-11-04 section: "Decisions" --- # Which AI bots and agents to let in | Am I Ready for Agents? > Which AI bots and agents to let in: the decision, three options with what each one costs, dated facts and the sentence to send your web team. Source: https://amireadyforagents.com/decisions/which-bots-to-allow Site: Am I Ready for Agents? Last reviewed: 2026-11-04 # Which AI bots and agents to let in ## In short Treat AI visitors as four kinds: training crawlers, AI search crawlers, assistants that fetch a page for a person and agents that act for a customer. Decide each on its own, write the answer down and make robots.txt and your bot protection say the same thing. Three common positions and their costs are below. ## The decision Decide, as four separate choices, whether you welcome training crawlers, AI search crawlers, assistants that fetch a page for a person, and agents that act for a customer, then make robots.txt and bot protection say the same thing. Who decides: Leadership decides the position; the web or security team enforces it and reports back. ## The options 1. **Allow all four kinds**Your content can train models, appear in AI answers and be read live for customers; you give up control over training use. [Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 4 Nov 2026. 2. **Refuse training, allow search, assistants and agents**The common middle path: 17 percent of Cloudflare sites block training while fewer than 1 percent block search. [Cloudflare: The Internet has a second audience (30 Sep 2026)](https://blog.cloudflare.com/agentic-web/), read 4 Nov 2026. 3. **Block assistants and agents too**Ora's lab found agents still answer when blocked, from other sources, and those answers miss more facts. [Ora research: AX is the new AEO (8 Oct 2026)](https://ora.ai/blog/ax-is-the-new-aeo?utm_source=amireadyforagents.com&utm_medium=referral&utm_campaign=decision), read 4 Nov 2026. ## What we found - 4 Nov 2026 In the network's own crawl, run of 4 Nov 2026 (13,071 of 13,071 domains from Tranco list 746WX plus Ora leaderboard domains), 13.9 percent of 8,624 sites with a readable robots.txt disallowed GPTBot at the root, against 6.9 percent for ChatGPT-User. [Scorecard methodology: the crawl of 4 Nov 2026](https://amireadyforagents.com/scorecards/methodology), read 4 Nov 2026 - 1 Jul 2026 Cloudflare groups AI traffic into Search, Agent and Training and lets every plan, including Free, block each on all pages, on pages with ads only, or not at all (changelog of 1 July 2026). [Cloudflare changelog: New options to manage AI traffic (1 Jul 2026)](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/), read 4 Nov 2026 - 1 Sep 2022 RFC 9309 states that robots.txt rules are not a form of access authorization; enforcement needs bot management. [RFC 9309 Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309), read 4 Nov 2026 ## How to decide - Start from the customer. If an agent acting for a buyer cannot reach your prices, that buyer's task goes to a competitor. - Refusing training while allowing the other three is the middle path in the options below. Decide it separately from search. - Ask who last looked at the CDN setting. A rule switched on for security can undo a decision made in marketing. - Write the position on one page with a review date, then ask the web team to confirm in writing that robots.txt and bot protection match it. ## Where this shows up - [Self-check question 1](https://amireadyforagents.com/self-check#q1)Has someone in your business deliberately decided which AI crawlers and assistants your robots.txt file allows, rather than leaving the default? - [Self-check question 2](https://amireadyforagents.com/self-check#q2)Has anyone checked in the last six months whether your CDN, firewall or bot protection blocks AI crawlers or AI assistants? - [The publishing guide](https://amireadyforagents.com/for/publishers)Separate controls for training, search and live fetches, and what Cloudflare changed in 2026. - [The software guide](https://amireadyforagents.com/for/saas)Agents that read docs and pricing, sign up for trials and open paid accounts. - [The local business guide](https://amireadyforagents.com/for/local)Google calling stores, searching reservation platforms and pointing customers to the provider. ## Sources 1. [Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) (blog.cloudflare.com/accountable-mixed-use-ai-crawlers), read 4 Nov 2026. 2. [Cloudflare: The Internet has a second audience (30 Sep 2026)](https://blog.cloudflare.com/agentic-web/) (blog.cloudflare.com/agentic-web), read 4 Nov 2026. 3. [Ora research: AX is the new AEO (8 Oct 2026)](https://ora.ai/blog/ax-is-the-new-aeo?utm_source=amireadyforagents.com&utm_medium=referral&utm_campaign=decision) (ora.ai/blog/ax-is-the-new-aeo), read 4 Nov 2026. 4. [Scorecard methodology: the crawl of 4 Nov 2026](https://amireadyforagents.com/scorecards/methodology) (amireadyforagents.com/scorecards/methodology), read 4 Nov 2026. 5. [Cloudflare changelog: New options to manage AI traffic (1 Jul 2026)](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/) (developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options), read 4 Nov 2026. 6. [RFC 9309 Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (rfc-editor.org/rfc/rfc9309), read 4 Nov 2026. 7. [Sector scorecards dataset, edition 4 Nov 2026](https://amireadyforagents.com/scorecards) (amireadyforagents.com/scorecards), read 4 Nov 2026. Last reviewed 4 November 2026 Reviewed 4 November 2026. [7 sources](#sources), each with the day it was read. ## Send this to your web team **Take this to your web team**Please export the current AI bot settings from our CDN or bot protection and tell us, for each of Search, Agent and Training traffic, whether it is allowed, challenged or blocked, and on which pages. The full list of sentences is on [ask your web team](https://amireadyforagents.com/ask-your-web-team).