Cloudflare syncs AI bot choices into robots.txt
In short
Cloudflare syncs AI bot choices into robots.txt, on 21 Aug 2026. The preference you state in robots.txt and the rule your CDN enforces can now be kept in step from one setting. Check whether your CDN offers the same and whether the two currently agree.
What happened
21 August 2026. Cloudflare launched Bot Preference Sync, which writes the choices a site sets for Search, Agent and Training traffic into its robots.txt so the stated preference and the enforced rule match. It is available from the Free plan up.
- 21 Aug 2026
Cloudflare launched Bot Preference Sync, which writes the choices a site sets for Search, Agent and Training traffic into its robots.txt so the stated preference and the enforced rule match. It is available from the Free plan up.
Why it matters to a leader
The preference you state in robots.txt and the rule your CDN enforces can now be kept in step from one setting.
What to check
Check whether your CDN offers the same and whether the two currently agree.
Sources
- Cloudflare: Bot Preference Sync (21 Aug 2026) (blog.cloudflare.com/bot-preference-sync), read 4 Nov 2026.
Last reviewed