Training crawler
In short
Refusing a training crawler is a separate choice from allowing search and from allowing assistants. Which AI bots and agents to let in is the next stop after Training crawler. Cloudflare reported 17 percent of its sites block training crawlers (30 Sep 2026).
Definition
A crawler that takes pages to train or fine-tune an AI model. Most providers let you refuse it separately from search.
A dated example
- Read 4 Nov 2026
Cloudflare reported 17 percent of its sites block training crawlers (30 Sep 2026).
Where it matters
Refusing it is a separate choice from allowing search and from allowing assistants.
Related terms
- AI search crawlerA crawler that indexes pages so an AI search product can answer questions and cite them later.
- Mixed-use crawlerOne crawler that does both search indexing and AI training, so refusing one used to mean refusing the other.
Sources
- Cloudflare: The Internet has a second audience (30 Sep 2026) (blog.cloudflare.com/agentic-web), read 4 Nov 2026.
Last reviewed