Mixed-use crawler
In short
A mixed-use crawler is why refusing training used to mean leaving search, and why new settings matter. Read on in What Cloudflare's AI defaults mean. Training crawler and AI search crawler sit beside it in the glossary. Cloudflare's Disallow AI Training setting of 15 Sep 2026 keeps Apple, Google and Microsoft crawlers allowed for search while refusing training.
Definition
One crawler that does both search indexing and AI training, so refusing one used to mean refusing the other.
A dated example
- Read 4 Nov 2026
Cloudflare's Disallow AI Training setting of 15 Sep 2026 keeps Apple, Google and Microsoft crawlers allowed for search while refusing training.
Where it matters
It is why refusing training used to mean leaving search, and why new settings matter.
Related terms
- Training crawlerA crawler that takes pages to train or fine-tune an AI model.
- AI search crawlerA crawler that indexes pages so an AI search product can answer questions and cite them later.
Sources
- Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026) (blog.cloudflare.com/accountable-mixed-use-ai-crawlers), read 4 Nov 2026.
Last reviewed