---
title: "robots.txt and bot rules | Am I Ready for Agents?"
description: "robots.txt and bot rules in plain terms: asking versus enforcing, the names of AI visitors and what Cloudflare changed in 2026."
canonical: https://amireadyforagents.com/university/robots-txt-and-bot-rules
site: "Am I Ready for Agents?"
last_updated: 2026-11-04
section: "Learn"
---

# robots.txt and bot rules | Am I Ready for Agents?

> robots.txt and bot rules in plain terms: asking versus enforcing, the names of AI visitors and what Cloudflare changed in 2026.

Source: https://amireadyforagents.com/university/robots-txt-and-bot-rules
Site: Am I Ready for Agents?
Last reviewed: 2026-11-04

# robots.txt and bot rules: asking versus enforcing

## In short

robots.txt is a public file that asks automated visitors to stay out of parts of your site. It is a request, not a lock. Enforcement happens at your CDN or bot protection, which is where 2026 changed the most: Cloudflare now separates Search, Agent and Training traffic and sets new defaults. Make the two say the same thing.

## Asking versus enforcing

robots.txt lists, by visitor name, the parts of your site a visitor may or may not fetch. Well-behaved crawlers follow it. The standard says it is not access control. Enforcement is the job of bot management at your CDN, host or firewall.

-   1 Sep 2022
    
    RFC 9309 states that robots.txt rules are not a form of access authorization; enforcement needs bot management.
    
    [RFC 9309 Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309), read 4 Nov 2026
    

## What changed in 2026 on Cloudflare

-   15 Sep 2026
    
    Cloudflare lets sites refuse AI training while staying in search. Cloudflare shipped a Disallow AI Training setting and an Accountable designation for Apple, Google and Microsoft crawlers, which stay allowed for search while training is refused. From the same date, new domains block Training and Agent bots on pages with ads by default.
    
    [Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 4 Nov 2026
    
-   21 Aug 2026
    
    Cloudflare syncs AI bot choices into robots.txt. Cloudflare launched Bot Preference Sync, which writes the choices a site sets for Search, Agent and Training traffic into its robots.txt so the stated preference and the enforced rule match. It is available from the Free plan up.
    
    [Cloudflare: Bot Preference Sync (21 Aug 2026)](https://blog.cloudflare.com/bot-preference-sync/), read 4 Nov 2026
    
-   1 Jul 2026
    
    Cloudflare groups AI traffic into Search, Agent and Training and lets every plan, including Free, block each on all pages, on pages with ads only, or not at all (changelog of 1 July 2026).
    
    [Cloudflare changelog: New options to manage AI traffic (1 Jul 2026)](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/), read 4 Nov 2026
    

If your site runs on another CDN or host, ask what its equivalent settings are and what is on by default.

## Names to know

Rules use the visitor's published name. GPTBot gathers material to train OpenAI models; ChatGPT-User fetches a page because a person asked a question. The two need different decisions. The [glossary](https://amireadyforagents.com/university/glossary) explains each kind, and the [decision page](https://amireadyforagents.com/decisions/which-bots-to-allow) sets out the options.

## Related

-   [Which AI bots to let in](https://amireadyforagents.com/decisions/which-bots-to-allow)The decision.
-   [What Cloudflare's AI defaults mean](https://amireadyforagents.com/guides/what-cloudflare-ai-defaults-mean-for-you)What to check today.

## Sources

1.  [RFC 9309 Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (rfc-editor.org/rfc/rfc9309), read 4 Nov 2026.
2.  [Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) (blog.cloudflare.com/accountable-mixed-use-ai-crawlers), read 4 Nov 2026.
3.  [Cloudflare: Bot Preference Sync (21 Aug 2026)](https://blog.cloudflare.com/bot-preference-sync/) (blog.cloudflare.com/bot-preference-sync), read 4 Nov 2026.
4.  [Cloudflare changelog: New options to manage AI traffic (1 Jul 2026)](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/) (developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options), read 4 Nov 2026.

Last reviewed 4 November 2026

Reviewed 4 November 2026. [4 sources](#sources), each with the day it was read.