---
title: "Training crawler: plain-language definition | Am I Ready for Agents?"
description: "Training crawler: A crawler that takes pages to train or fine-tune an AI model. What it is, a dated example and where it matters. Dated and sourced."
canonical: https://amireadyforagents.com/glossary/training-crawler
site: "Am I Ready for Agents?"
last_updated: 2026-11-04
section: "Glossary"
---

# Training crawler: plain-language definition | Am I Ready for Agents?

> Training crawler: A crawler that takes pages to train or fine-tune an AI model. What it is, a dated example and where it matters. Dated and sourced.

Source: https://amireadyforagents.com/glossary/training-crawler
Site: Am I Ready for Agents?
Last reviewed: 2026-11-04

# Training crawler

## In short

Refusing a training crawler is a separate choice from allowing search and from allowing assistants. Which AI bots and agents to let in is the next stop after Training crawler. Cloudflare reported 17 percent of its sites block training crawlers (30 Sep 2026).

## Definition

A crawler that takes pages to train or fine-tune an AI model. Most providers let you refuse it separately from search.

## A dated example

-   Read 4 Nov 2026
    
    Cloudflare reported 17 percent of its sites block training crawlers (30 Sep 2026).
    
    [Cloudflare: The Internet has a second audience (30 Sep 2026)](https://blog.cloudflare.com/agentic-web/), read 4 Nov 2026
    

## Where it matters

Refusing it is a separate choice from allowing search and from allowing assistants.

-   [Which AI bots and agents to let in](https://amireadyforagents.com/decisions/which-bots-to-allow)

## Related terms

-   [AI search crawler](https://amireadyforagents.com/glossary/ai-search-crawler)A crawler that indexes pages so an AI search product can answer questions and cite them later.
-   [Mixed-use crawler](https://amireadyforagents.com/glossary/mixed-use-crawler)One crawler that does both search indexing and AI training, so refusing one used to mean refusing the other.

## Sources

1.  [Cloudflare: The Internet has a second audience (30 Sep 2026)](https://blog.cloudflare.com/agentic-web/) (blog.cloudflare.com/agentic-web), read 4 Nov 2026.

Last reviewed 4 November 2026

Reviewed 4 November 2026. [1 source](#sources), each with the day it was read.