Am I Ready for Agents?

Pay per crawl

In short

Pay per crawl is the option behind charging AI crawlers, a trade of revenue against reach. Whether to charge AI crawlers and agents goes into the detail. The neighbouring terms are AI crawler and x402. Cloudflare's AI Crawl Control documentation includes setting a pay per crawl price (read 4 Nov 2026).

Definition

Charging AI crawlers a fee for each page they fetch, collected through your CDN.

A dated example

  • Read 4 Nov 2026

    Cloudflare's AI Crawl Control documentation includes setting a pay per crawl price (read 4 Nov 2026).

    Cloudflare AI Crawl Control docs, read 4 Nov 2026

Where it matters

It is the option behind charging AI crawlers, a trade of revenue against reach.

  • AI crawlerA bot that collects pages in bulk on its own schedule, to train a model or build an AI search index.
  • x402A way for a site to ask for payment inside the web request itself, so an agent can pay a small amount and retry without an account.

Sources

  1. Cloudflare AI Crawl Control docs (developers.cloudflare.com/ai-crawl-control), read 4 Nov 2026.

Last reviewed