AI crawler
In short
An AI crawler gathers pages on its own schedule, so blocking one changes what an AI system knows about you later, not a customer's request today. Cloudflare reported that 52 percent of crawler requests it saw in June 2026 were for AI training by stated purpose.
Definition
A bot that collects pages in bulk on its own schedule, to train a model or build an AI search index. Blocking it changes what an AI system knows about you later, not a customer's request now.
A dated example
- Read 4 Nov 2026
Cloudflare reported that 52 percent of crawler requests it saw in June 2026 were for AI training by stated purpose.
Where it matters
Blocking one changes what an AI system knows about you later, not a customer's request today.
Related terms
- Training crawlerA crawler that takes pages to train or fine-tune an AI model.
- AI search crawlerA crawler that indexes pages so an AI search product can answer questions and cite them later.
- Mixed-use crawlerOne crawler that does both search indexing and AI training, so refusing one used to mean refusing the other.
Sources
- Cloudflare: The Internet has a second audience (30 Sep 2026) (blog.cloudflare.com/agentic-web), read 4 Nov 2026.
Last reviewed