Cloudflare (AI Crawl Control, Markdown for Agents, pay per crawl): what it does and what it costs
In short
Cloudflare, the network in front of many business sites, shows which AI crawlers reach a site and lets the owner allow or block each one, on every plan including Free. Markdown for Agents is included on paid plans at no extra cost, and pay per crawl, which charges crawlers, is a closed beta.
At a glance
- What it is
- The CDN in front of many business sites, which shows and controls which AI crawlers reach a site on every plan, serves Markdown to agents on paid plans, and lets sites charge crawlers in a closed beta.
- Type
- Web platform
- Operator
- Cloudflare
- Homepage
- cloudflare.com
- Homepage check
- HTTP 200 on 5 November 2026
- Last read
- 5 November 2026
What it does for your business
- Read 5 Nov 2026
Cloudflare's docs say AI Crawl Control, formerly AI Audit, is available on all plans and shows which AI services access a site's content.
- Read 5 Nov 2026
Cloudflare's docs say Markdown for Agents is available to Pro, Business and Enterprise plans and SSL for SaaS customers at no cost.
- Read 5 Nov 2026
Cloudflare's docs say pay per crawl is in closed beta and lets a site set a price per zone, answering unpaid AI crawlers with HTTP 402 and the price.
- Read 5 Nov 2026
Cloudflare's docs say Cloudflare acts as merchant of record for pay per crawl.
Detail for your web team
| Read | Fact | Source |
|---|---|---|
| 5 Nov 2026 | Cloudflare's docs say AI Crawl Control can allow or block individual crawlers and track which crawlers follow robots.txt. | AI Crawl Control (Cloudflare docs) |
| 5 Nov 2026 | Cloudflare's docs say Markdown for Agents converts a page's HTML to Markdown at the edge when an AI system asks for text/markdown. | Markdown for Agents (Cloudflare docs) |
| 5 Nov 2026 | Cloudflare's docs say Markdown for Agents adds a default Content-Signal of ai-train=yes, search=yes, ai-input=yes when the origin sends no content-signal header of its own. | Markdown for Agents (Cloudflare docs) |
| 5 Nov 2026 | Cloudflare's docs say Markdown for Agents converts at most 6 MiB of decompressed HTML per page. | Markdown for Agents (Cloudflare docs) |
| 5 Nov 2026 | Cloudflare's docs say a block in Cloudflare's WAF or Bot Management overrides a pay per crawl charge, so the blocked crawler gets no access. | What is Pay Per Crawl? (Cloudflare docs) |
Products and plans
| Product | What it is | Price | As of | Source |
|---|---|---|---|---|
| AI Crawl Control | Dashboard and rules for AI crawler access. | Included in all plans, 0 USD. Available on all plans including Free. | 5 Nov 2026 | AI Crawl Control (Cloudflare docs) |
| Markdown for Agents | Edge conversion of HTML to Markdown for agents that ask for it. | Included in paid plans. Pro, Business, Enterprise and SSL for SaaS at no extra cost. | 5 Nov 2026 | Markdown for Agents (Cloudflare docs) |
| Pay per crawl | Charge AI crawlers a per-zone price through HTTP 402. | Closed beta | 5 Nov 2026 | What is Pay Per Crawl? (Cloudflare docs) |
Standards it states
| Standard | Status | What the page says | Source |
|---|---|---|---|
| Markdown for agents | Supports | Markdown for Agents answers Accept: text/markdown. | Markdown for Agents (Cloudflare docs) |
| AI bot rules in robots.txt | Supports | AI Crawl Control tracks robots.txt compliance and enforces rules. | AI Crawl Control (Cloudflare docs) |
| Content Signals | Supports | Markdown for Agents sets or preserves Content-Signal headers. | Markdown for Agents (Cloudflare docs) |
The site's own checks
| Check | Domain | Result | Date | Method | Source |
|---|---|---|---|---|---|
| Cloudflare isitagentready.com keyless scan | cloudflare.com | 60 of 100, L4 Agent-Integrated, 9 of 15 counted checks passed | Run 5 Nov 2026 | POST https://isitagentready.com/api/scan, default All Checks profile, score = passed / counted x 100 | Source |
| Ora cached score | cloudflare.com | 82 of 100, grade B | Scanned 5 Nov 2026, read 5 Nov 2026 | GET /api/score (cached) | Source |
What it does not do
- Cloudflare does not offer Markdown for Agents on the Free plan.
- Markdown for Agents does not withhold training permission by default; without an origin header it signals ai-train=yes.
- Pay per crawl is not open to every site; it is a closed beta.
Related pages on this site
Sources
- AI Crawl Control (Cloudflare docs) (developers.cloudflare.com/ai-crawl-control/index.md), read 5 Nov 2026.
- Markdown for Agents (Cloudflare docs) (developers.cloudflare.com/fundamentals/reference/markdown-for-agents/index.md), read 5 Nov 2026.
- What is Pay Per Crawl? (Cloudflare docs) (developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/index.md), read 5 Nov 2026.
- Is Your Site Agent-Ready? (Cloudflare, isitagentready.com) (isitagentready.com), read 5 Nov 2026.
- Ora cached score for cloudflare.com (ora.ai/api/score/cloudflare.com), read 5 Nov 2026.
Last reviewed