---
title: "Mixed-use crawler: plain-language definition | Am I Ready for Agents?"
description: "Mixed-use crawler: One crawler that does both search indexing and AI training, so refusing one used to mean refusing the other. Dated and sourced."
canonical: https://amireadyforagents.com/glossary/mixed-use-crawler
site: "Am I Ready for Agents?"
last_updated: 2026-11-04
section: "Glossary"
---

# Mixed-use crawler: plain-language definition | Am I Ready for Agents?

> Mixed-use crawler: One crawler that does both search indexing and AI training, so refusing one used to mean refusing the other. Dated and sourced.

Source: https://amireadyforagents.com/glossary/mixed-use-crawler
Site: Am I Ready for Agents?
Last reviewed: 2026-11-04

# Mixed-use crawler

## In short

A mixed-use crawler is why refusing training used to mean leaving search, and why new settings matter. Read on in What Cloudflare's AI defaults mean. Training crawler and AI search crawler sit beside it in the glossary. Cloudflare's Disallow AI Training setting of 15 Sep 2026 keeps Apple, Google and Microsoft crawlers allowed for search while refusing training.

## Definition

One crawler that does both search indexing and AI training, so refusing one used to mean refusing the other.

## A dated example

-   Read 4 Nov 2026
    
    Cloudflare's Disallow AI Training setting of 15 Sep 2026 keeps Apple, Google and Microsoft crawlers allowed for search while refusing training.
    
    [Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 4 Nov 2026
    

## Where it matters

It is why refusing training used to mean leaving search, and why new settings matter.

-   [What Cloudflare's AI defaults mean](https://amireadyforagents.com/guides/what-cloudflare-ai-defaults-mean-for-you)

## Related terms

-   [Training crawler](https://amireadyforagents.com/glossary/training-crawler)A crawler that takes pages to train or fine-tune an AI model.
-   [AI search crawler](https://amireadyforagents.com/glossary/ai-search-crawler)A crawler that indexes pages so an AI search product can answer questions and cite them later.

## Sources

1.  [Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) (blog.cloudflare.com/accountable-mixed-use-ai-crawlers), read 4 Nov 2026.

Last reviewed 4 November 2026

Reviewed 4 November 2026. [1 source](#sources), each with the day it was read.