Am I Ready for Agents?

Whether to charge AI crawlers and agents

In short

Charging AI crawlers and agents for access can bring revenue from traffic that no longer clicks through, and it can remove you from the answers of operators who decline to pay. Three options are below, from not charging to blocking unless paid, with dated facts and a way to test on one section first.

The decision

Decide whether to ask AI operators to pay for access to some or all of your content, accepting that some will decline and stop citing you.

Who decides: The commercial lead and the publisher decide; the web team runs the test.

The options

  1. Do not chargeSimplest; your reach in AI answers depends only on your allow and block rules. Cloudflare: The Internet has a second audience (30 Sep 2026), read 4 Nov 2026.
  2. Charge on a test sectionLets you measure revenue against lost reach before committing. Cloudflare describes per-fetch charging and read-time licensing as the model it is building toward. Cloudflare: the agentic internet, WebMCP example (6 Aug 2026), read 4 Nov 2026.
  3. Block unless paidStrongest control; operators that do not pay lose access and your pages drop out of their answers. Cloudflare: the agentic internet, WebMCP example (6 Aug 2026), read 4 Nov 2026.

What we found

How to decide

  • Test on one section of the site, not the whole site, and name the reach metric before you start.
  • Keep charging separate from blocking. A site can allow search and still charge for other uses.
  • Agree who counts the revenue and who owns the loss of reach, so the review is not an argument about whose number is right.

Where this shows up

  • Self-check question 1Has someone in your business deliberately decided which AI crawlers and assistants your robots.txt file allows, rather than leaving the default?
  • The publishing guideSeparate controls for training, search and live fetches, and what Cloudflare changed in 2026.

Sources

  1. Cloudflare: The Internet has a second audience (30 Sep 2026) (blog.cloudflare.com/agentic-web), read 4 Nov 2026.
  2. Cloudflare: the agentic internet, WebMCP example (6 Aug 2026) (blog.cloudflare.com/the-agentic-internet), read 4 Nov 2026.
  3. Cloudflare AI Crawl Control docs (developers.cloudflare.com/ai-crawl-control), read 4 Nov 2026.

Last reviewed