robots.txt
In short
robots.txt is where you state your position in public, and it works only if your bot protection matches it. robots.txt and bot rules goes into the detail. User agent and Content Signals and Bot management sit beside it in the glossary. RFC 9309 says robots.txt rules are not a form of access authorization.
Definition
A public text file at the root of a site that tells automated visitors which parts they may visit. It is a request, not a lock.
A dated example
- Read 4 Nov 2026
RFC 9309 says robots.txt rules are not a form of access authorization.
Where it matters
It is where you state your position in public, and it works only if your bot protection matches it.
Related terms
- User agentThe name a browser or bot gives itself with each request.
- Content SignalsA line in robots.txt that states how your content may be used, for example for search, AI input or training.
- Bot managementTools at your CDN, host or firewall that identify automated visitors and allow, challenge or block them.
Sources
- RFC 9309 Robots Exclusion Protocol (rfc-editor.org/rfc/rfc9309), read 4 Nov 2026.
Last reviewed