---
title: "Whether to charge AI crawlers | Am I Ready for Agents?"
description: "Whether to charge AI crawlers and agents: the decision, three options with what each one costs, dated facts and the sentence to send your web team."
canonical: https://amireadyforagents.com/decisions/whether-to-charge-crawlers
site: "Am I Ready for Agents?"
last_updated: 2026-11-04
section: "Decisions"
---

# Whether to charge AI crawlers | Am I Ready for Agents?

> Whether to charge AI crawlers and agents: the decision, three options with what each one costs, dated facts and the sentence to send your web team.

Source: https://amireadyforagents.com/decisions/whether-to-charge-crawlers
Site: Am I Ready for Agents?
Last reviewed: 2026-11-04

# Whether to charge AI crawlers and agents

## In short

Charging AI crawlers and agents for access can bring revenue from traffic that no longer clicks through, and it can remove you from the answers of operators who decline to pay. Three options are below, from not charging to blocking unless paid, with dated facts and a way to test on one section first.

## The decision

Decide whether to ask AI operators to pay for access to some or all of your content, accepting that some will decline and stop citing you.

Who decides: The commercial lead and the publisher decide; the web team runs the test.

## The options

1.  **Do not charge**Simplest; your reach in AI answers depends only on your allow and block rules. [Cloudflare: The Internet has a second audience (30 Sep 2026)](https://blog.cloudflare.com/agentic-web/), read 4 Nov 2026.
2.  **Charge on a test section**Lets you measure revenue against lost reach before committing. Cloudflare describes per-fetch charging and read-time licensing as the model it is building toward. [Cloudflare: the agentic internet, WebMCP example (6 Aug 2026)](https://blog.cloudflare.com/the-agentic-internet/), read 4 Nov 2026.
3.  **Block unless paid**Strongest control; operators that do not pay lose access and your pages drop out of their answers. [Cloudflare: the agentic internet, WebMCP example (6 Aug 2026)](https://blog.cloudflare.com/the-agentic-internet/), read 4 Nov 2026.

## What we found

-   30 Sep 2026
    
    Cloudflare reported on 30 September 2026 that human traffic in heavily crawled categories such as Retail and Financial Services fell as much as 40 percent in under a year.
    
    [Cloudflare: The Internet has a second audience (30 Sep 2026)](https://blog.cloudflare.com/agentic-web/), read 4 Nov 2026
    
-   6 Aug 2026
    
    Cloudflare said on 6 August 2026 that a recipe site could charge a fraction of a cent per fetch, and that each paid interaction leaves a receipt both sides can check.
    
    [Cloudflare: the agentic internet, WebMCP example (6 Aug 2026)](https://blog.cloudflare.com/the-agentic-internet/), read 4 Nov 2026
    
-   Read 4 Nov 2026
    
    Cloudflare's AI Crawl Control documentation lists pay per crawl among its controls (read 4 Nov 2026).
    
    [Cloudflare AI Crawl Control docs](https://developers.cloudflare.com/ai-crawl-control/), read 4 Nov 2026
    

## How to decide

-   Test on one section of the site, not the whole site, and name the reach metric before you start.
-   Keep charging separate from blocking. A site can allow search and still charge for other uses.
-   Agree who counts the revenue and who owns the loss of reach, so the review is not an argument about whose number is right.

## Where this shows up

-   [Self-check question 1](https://amireadyforagents.com/self-check#q1)Has someone in your business deliberately decided which AI crawlers and assistants your robots.txt file allows, rather than leaving the default?
-   [The publishing guide](https://amireadyforagents.com/for/publishers)Separate controls for training, search and live fetches, and what Cloudflare changed in 2026.

## Sources

1.  [Cloudflare: The Internet has a second audience (30 Sep 2026)](https://blog.cloudflare.com/agentic-web/) (blog.cloudflare.com/agentic-web), read 4 Nov 2026.
2.  [Cloudflare: the agentic internet, WebMCP example (6 Aug 2026)](https://blog.cloudflare.com/the-agentic-internet/) (blog.cloudflare.com/the-agentic-internet), read 4 Nov 2026.
3.  [Cloudflare AI Crawl Control docs](https://developers.cloudflare.com/ai-crawl-control/) (developers.cloudflare.com/ai-crawl-control), read 4 Nov 2026.

Last reviewed 4 November 2026

Reviewed 4 November 2026. [3 sources](#sources), each with the day it was read.

## Send this to your web team

**Take this to your web team**Please set up a paid-access test on one named section, report AI crawler and agent requests by operator every month, and tell us which operators stopped fetching during the test.

The full list of sentences is on [ask your web team](https://amireadyforagents.com/ask-your-web-team).