Am I Ready for Agents?

robots.txt and bot rules: asking versus enforcing

In short

robots.txt is a public file that asks automated visitors to stay out of parts of your site. It is a request, not a lock. Enforcement happens at your CDN or bot protection, which is where 2026 changed the most: Cloudflare now separates Search, Agent and Training traffic and sets new defaults. Make the two say the same thing.

Asking versus enforcing

robots.txt lists, by visitor name, the parts of your site a visitor may or may not fetch. Well-behaved crawlers follow it. The standard says it is not access control. Enforcement is the job of bot management at your CDN, host or firewall.

  • 1 Sep 2022

    RFC 9309 states that robots.txt rules are not a form of access authorization; enforcement needs bot management.

    RFC 9309 Robots Exclusion Protocol, read 4 Nov 2026

What changed in 2026 on Cloudflare

  • 15 Sep 2026

    Cloudflare lets sites refuse AI training while staying in search. Cloudflare shipped a Disallow AI Training setting and an Accountable designation for Apple, Google and Microsoft crawlers, which stay allowed for search while training is refused. From the same date, new domains block Training and Agent bots on pages with ads by default.

    Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026), read 4 Nov 2026
  • 21 Aug 2026

    Cloudflare syncs AI bot choices into robots.txt. Cloudflare launched Bot Preference Sync, which writes the choices a site sets for Search, Agent and Training traffic into its robots.txt so the stated preference and the enforced rule match. It is available from the Free plan up.

    Cloudflare: Bot Preference Sync (21 Aug 2026), read 4 Nov 2026
  • 1 Jul 2026

    Cloudflare groups AI traffic into Search, Agent and Training and lets every plan, including Free, block each on all pages, on pages with ads only, or not at all (changelog of 1 July 2026).

    Cloudflare changelog: New options to manage AI traffic (1 Jul 2026), read 4 Nov 2026

If your site runs on another CDN or host, ask what its equivalent settings are and what is on by default.

Names to know

Rules use the visitor's published name. GPTBot gathers material to train OpenAI models; ChatGPT-User fetches a page because a person asked a question. The two need different decisions. The glossary explains each kind, and the decision page sets out the options.

Sources

  1. RFC 9309 Robots Exclusion Protocol (rfc-editor.org/rfc/rfc9309), read 4 Nov 2026.
  2. Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026) (blog.cloudflare.com/accountable-mixed-use-ai-crawlers), read 4 Nov 2026.
  3. Cloudflare: Bot Preference Sync (21 Aug 2026) (blog.cloudflare.com/bot-preference-sync), read 4 Nov 2026.
  4. Cloudflare changelog: New options to manage AI traffic (1 Jul 2026) (developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options), read 4 Nov 2026.

Last reviewed