AI search crawler
In short
Blocking an AI search crawler removes you from the index an AI answer is built from, so decide it deliberately. Which AI bots and agents to let in goes into the detail. Compare it with Training crawler and Mixed-use crawler. Cloudflare reported fewer than 1 percent of its sites block search crawlers (30 Sep 2026).
Definition
A crawler that indexes pages so an AI search product can answer questions and cite them later.
A dated example
- Read 4 Nov 2026
Cloudflare reported fewer than 1 percent of its sites block search crawlers (30 Sep 2026).
Where it matters
Blocking it removes you from the index an AI answer is built from, so decide it deliberately.
Related terms
- Training crawlerA crawler that takes pages to train or fine-tune an AI model.
- Mixed-use crawlerOne crawler that does both search indexing and AI training, so refusing one used to mean refusing the other.
Sources
- Cloudflare: The Internet has a second audience (30 Sep 2026) (blog.cloudflare.com/agentic-web), read 4 Nov 2026.
Last reviewed