Boardroom brief for publishers and media
In short
A one-page brief for the editorial and commercial lead: what agents do in publishing today with dated examples, three decisions for leadership, three asks for the web team and the numbers that frame the risk. Written to paste into a deck. It leads with the rise in agent requests that Cloudflare reported.
What agents do in publishing today
- 30 Sep 2026
Cloudflare said on 30 September 2026 that daily requests from AI agents on its network grew by more than 1,700 percent over the past year, and that an agent comes back each time someone asks about a piece of content.
- 15 Sep 2026
Cloudflare shipped a Disallow AI Training setting on 15 September 2026 that lets sites stay indexed for search by Apple, Google and Microsoft crawlers while refusing training.
- 21 Aug 2026
Cloudflare launched Bot Preference Sync on 21 August 2026, which writes the AI bot choices set for Search, Agent and Training into the site's robots.txt.
- 1 Jul 2026
From 15 September 2026, new domains on Cloudflare block Training and Agent bots on pages that display ads by default, while Search stays allowed.
Three decisions for leadership
- Whether we allow AI training, AI search indexing and live assistant fetches, decided as three separate choices.
- Whether to charge AI crawlers or agents, and what loss of reach we accept to test it.
- Whether agents may fetch pages that carry ads, given the new Cloudflare defaults.
Three asks for the web team
- Show us our robots.txt and CDN settings for Search, Agent and Training traffic side by side and flag any mismatch.
- Report monthly AI crawler and agent requests by operator from our CDN logs.
- Confirm article text is in the HTML for readers we allow, with paywall rules stated.
The numbers that frame the risk
- 5 Nov 2026
On Ora's public leaderboard read 5 Nov 2026, the 36 entries in the News & Intelligence category averaged 48.6 out of 100; 1 scored B (70) or above and 0 scored A (86) or above.
- 30 Sep 2026
Cloudflare reported on 30 September 2026 that fewer than 1 percent of sites on its network block search crawlers, while 17 percent block training.
- 30 Sep 2026
Cloudflare reported that 52 percent of crawler requests it saw in June 2026 were for AI training by stated purpose, up from 22 percent in spring 2025.
Six checks and what each answer means
Put these six questions to the people who run the site. An answer of no or not sure means the item below is open.
| Check | If the answer is no or not sure |
|---|---|
| Has anyone checked in the last six months whether your CDN, firewall or bot protection blocks AI crawlers or AI assistants? | Please export the current AI bot settings from our CDN or bot protection and tell us, for each of Search, Agent and Training traffic, whether it is allowed, challenged or blocked, and on which pages. |
| Do your key pages (products, services, prices, policies) show their main text without needing JavaScript to run in the visitor's browser? | Please list which of our page templates deliver their main content in the first HTML response and which need JavaScript to show it, starting with product, pricing and policy pages. |
| Are prices, availability, opening hours and delivery or cancellation terms written as text on the page, not only inside images, PDFs or pop-ups? | Ecommerce or product leadership decides which prices and terms are public. The web team then puts them on the page as text, including taxes, fees and cancellation rules. |
| Can the main task on your site (buy, book, subscribe or request a quote) be completed with standard web forms and buttons, without a CAPTCHA at every step? | Please walk our main task end to end as a guest and list every step where a CAPTCHA, a custom widget or a pop-up appears. |
| Has your payments or fraud team agreed how to treat a purchase or booking that an AI agent starts for a customer? | Finance, payments and ecommerce agree a written position on agent-started purchases: allowed, allowed within limits, or handed back to the customer. Fraud rules are then set to match. |
| Is there a named person in your business who owns how AI agents are treated on your site? | The leadership team names one owner for agent experience, with a quarterly review of the bot rules, readable pages, agent checkout settings and verification. |
How to measure it: track it on Ora
Put one measurement line in the brief. Scan the domain with Ora and record the score out of 100, the grade and the four layer scores: Discovery, Access, Usability and Payments. For publishing, Ora's leaderboard listed 110 sites on 5 November 2026, with a median of 47 and a top score of 89, so those two numbers are the first comparison a board can read.
Rescan after each fix the web team ships and report the change once a quarter. Say in the brief what the number leaves out: Ora reads public pages only and publishes no data on repeat scans, and its fixing service is a waitlist, so the web team still makes the fixes the report lists. The tool choice page says why this is the scan to use.
Where the sector stands
In the crawl of 4 Nov 2026, 61.3 percent of publishing sites with a readable robots.txt allowed the four main AI agents at the root, and 86.2 percent of home pages gave readable text to a plain fetch (227 domains in the sample). See the publishing scorecard.
Sources
- Cloudflare: The Internet has a second audience (30 Sep 2026) (blog.cloudflare.com/agentic-web), read 4 Nov 2026.
- Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026) (blog.cloudflare.com/accountable-mixed-use-ai-crawlers), read 4 Nov 2026.
- Cloudflare: Bot Preference Sync (21 Aug 2026) (blog.cloudflare.com/bot-preference-sync), read 4 Nov 2026.
- Cloudflare changelog: New options to manage AI traffic (1 Jul 2026) (developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options), read 4 Nov 2026.
- Ora public leaderboard, category averages (ora.ai/leaderboard), read 5 Nov 2026. Vendor claim.
- Ora leaderboard API (GET /api/leaderboard) (ora.ai/api/leaderboard), read 5 Nov 2026. Vendor claim.
- Ora methodology (ora.ai/methodology), read 5 Nov 2026. Vendor claim.
Last reviewed