What AI agents mean for publishers and media

What agents do in publishing today

Written for the publisher, the head of audience and the commercial lead who decide what AI systems may do with the articles, and for the web lead who sets the CDN rules.

• 30 Sep 2026

  Cloudflare said on 30 September 2026 that daily requests from AI agents on its network grew by more than 1,700 percent over the past year, and that an agent comes back each time someone asks about a piece of content.

  Cloudflare: The Internet has a second audience (30 Sep 2026), read 4 Nov 2026
• 15 Sep 2026

  Cloudflare shipped a Disallow AI Training setting on 15 September 2026 that lets sites stay indexed for search by Apple, Google and Microsoft crawlers while refusing training.

  Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026), read 4 Nov 2026
• 21 Aug 2026

  Cloudflare launched Bot Preference Sync on 21 August 2026, which writes the AI bot choices set for Search, Agent and Training into the site's robots.txt.

  Cloudflare: Bot Preference Sync (21 Aug 2026), read 4 Nov 2026
• 14 Aug 2026

  Ora's report of 8 Oct 2026, citing Promptwatch, says Reddit's share of ChatGPT search citations fell from 3.83 percent to 0.52 percent in August 2026 as ChatGPT began running site: searches against official sites.

  Vendor claim. Ora research: AX is the new AEO (8 Oct 2026), read 4 Nov 2026
• 1 Jul 2026

  From 15 September 2026, new domains on Cloudflare block Training and Agent bots on pages that display ads by default, while Search stays allowed.

  Cloudflare changelog: New options to manage AI traffic (1 Jul 2026), read 4 Nov 2026

Decisions for leadership

These three belong to the editorial and commercial lead, not to the developers. Each can be settled in a meeting, and each changes what the web team is asked to build.

1. Whether we allow AI training, AI search indexing and live assistant fetches, decided as three separate choices.
2. Whether to charge AI crawlers or agents, and what loss of reach we accept to test it.
3. Whether agents may fetch pages that carry ads, given the new Cloudflare defaults.

• Which AI bots and agents to let inFour kinds of AI visitor, four separate choices, one robots.txt that matches your bot protection.
• Whether to charge AI crawlers and agentsRevenue against reach: what charging for access can cost you in AI answers.
• How to tell verified agents from unknown botsWhether agents that prove who runs them are treated differently from unknown bots.

The numbers that frame the risk

Figures labelled as vendor claims come from Ora or another vendor's own publication and are given with the date they were read. The crawl figures are this site's own.

• 5 Nov 2026

  On Ora's public leaderboard read 5 Nov 2026, the 36 entries in the News & Intelligence category averaged 48.6 out of 100; 1 scored B (70) or above and 0 scored A (86) or above.

  Vendor claim. Ora public leaderboard, category averages, read 5 Nov 2026
• 4 Nov 2026

  In the network's own crawl, run of 4 Nov 2026 (13,071 of 13,071 domains from Tranco list 746WX plus Ora leaderboard domains), 14.9 percent of 8,624 sites with a readable robots.txt disallowed CCBot and 13.9 percent disallowed GPTBot at the root; 7.4 percent published a Content-Signals line.

  Scorecard methodology: the crawl of 4 Nov 2026, read 4 Nov 2026
• 30 Sep 2026

  Cloudflare reported on 30 September 2026 that fewer than 1 percent of sites on its network block search crawlers, while 17 percent block training.

  Cloudflare: The Internet has a second audience (30 Sep 2026), read 4 Nov 2026
• 30 Sep 2026

  Cloudflare reported that 52 percent of crawler requests it saw in June 2026 were for AI training by stated purpose, up from 22 percent in spring 2025.

  Cloudflare: The Internet has a second audience (30 Sep 2026), read 4 Nov 2026

Where publishing stands on Ora's leaderboard

Ora ranks sites on a public leaderboard, grouped by category. On 5 November 2026 it listed 110 sites in the Gaming & Media and News & Intelligence categories, the ones that stand in for publishing in this site's crawl. The median score was 47 out of 100 and the top score was 89. 4 of the 110 scored B (70) or above. These are Ora's figures, vendor-published, and they describe the sites Ora lists, not every business in the sector.

The middle half of the listed sites scored between about 40 and 56. A score of 40 or lower sits in the bottom quarter of the publishing sites Ora lists, and 56 or higher in the top quarter. To place your own site, scan it with Ora (free, public pages only) and read the result against these numbers, then see how competitors in your sector score.

Where publishing stands in the crawl

• 61%allow the four main AI agents
  119 of 194 sites with a readable robots.txt let GPTBot, ClaudeBot, PerplexityBot and Google-Extended in at the root.
• 86%show readable text on the home page
  144 of 167 home pages gave a plain fetch 150 words or more of text.

The sample is 227 publishing domains from the crawl of 4 Nov 2026. The sector figures use Ora leaderboard categories as stand-ins, not a definition of the sector: Gaming & Media (74 rows), News & Intelligence (36 rows), plus 117 domains placed by a keyword on the domain name. The full publishing scorecard has all four measures and the method.

Check your own business

• Self-check for publishingThe 18 core questions with examples from this sector, plus two extra questions. Scored in your browser.
• Boardroom brief for publishingOne page you can paste into a deck, with the decisions and the numbers above.
• Scorecard for publishingThe four crawl measures for this sector and how they were counted.