---
title: "AI crawler: plain-language definition | Am I Ready for Agents?"
description: "AI crawler: A bot that collects pages in bulk on its own schedule, to train a model or build an AI search index. Dated and sourced."
canonical: https://amireadyforagents.com/glossary/ai-crawler
site: "Am I Ready for Agents?"
last_updated: 2026-11-04
section: "Glossary"
---

# AI crawler: plain-language definition | Am I Ready for Agents?

> AI crawler: A bot that collects pages in bulk on its own schedule, to train a model or build an AI search index. Dated and sourced.

Source: https://amireadyforagents.com/glossary/ai-crawler
Site: Am I Ready for Agents?
Last reviewed: 2026-11-04

# AI crawler

## In short

An AI crawler gathers pages on its own schedule, so blocking one changes what an AI system knows about you later, not a customer's request today. Cloudflare reported that 52 percent of crawler requests it saw in June 2026 were for AI training by stated purpose.

## Definition

A bot that collects pages in bulk on its own schedule, to train a model or build an AI search index. Blocking it changes what an AI system knows about you later, not a customer's request now.

## A dated example

-   Read 4 Nov 2026
    
    Cloudflare reported that 52 percent of crawler requests it saw in June 2026 were for AI training by stated purpose.
    
    [Cloudflare: The Internet has a second audience (30 Sep 2026)](https://blog.cloudflare.com/agentic-web/), read 4 Nov 2026
    

## Where it matters

Blocking one changes what an AI system knows about you later, not a customer's request today.

-   [Which AI bots and agents to let in](https://amireadyforagents.com/decisions/which-bots-to-allow)

## Related terms

-   [Training crawler](https://amireadyforagents.com/glossary/training-crawler)A crawler that takes pages to train or fine-tune an AI model.
-   [AI search crawler](https://amireadyforagents.com/glossary/ai-search-crawler)A crawler that indexes pages so an AI search product can answer questions and cite them later.
-   [Mixed-use crawler](https://amireadyforagents.com/glossary/mixed-use-crawler)One crawler that does both search indexing and AI training, so refusing one used to mean refusing the other.

## Sources

1.  [Cloudflare: The Internet has a second audience (30 Sep 2026)](https://blog.cloudflare.com/agentic-web/) (blog.cloudflare.com/agentic-web), read 4 Nov 2026.

Last reviewed 4 November 2026

Reviewed 4 November 2026. [1 source](#sources), each with the day it was read.