---
title: "robots.txt: plain-language definition | Am I Ready for Agents?"
description: "robots.txt: A public text file at the root of a site that tells automated visitors which parts they may visit. What it is, a dated example and where it matters."
canonical: https://amireadyforagents.com/glossary/robots-txt
site: "Am I Ready for Agents?"
last_updated: 2026-11-04
section: "Glossary"
---

# robots.txt: plain-language definition | Am I Ready for Agents?

> robots.txt: A public text file at the root of a site that tells automated visitors which parts they may visit. What it is, a dated example and where it matters.

Source: https://amireadyforagents.com/glossary/robots-txt
Site: Am I Ready for Agents?
Last reviewed: 2026-11-04

# robots.txt

## In short

robots.txt is where you state your position in public, and it works only if your bot protection matches it. robots.txt and bot rules goes into the detail. User agent and Content Signals and Bot management sit beside it in the glossary. RFC 9309 says robots.txt rules are not a form of access authorization.

## Definition

A public text file at the root of a site that tells automated visitors which parts they may visit. It is a request, not a lock.

## A dated example

-   Read 4 Nov 2026
    
    RFC 9309 says robots.txt rules are not a form of access authorization.
    
    [RFC 9309 Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309), read 4 Nov 2026
    

## Where it matters

It is where you state your position in public, and it works only if your bot protection matches it.

-   [robots.txt and bot rules](https://amireadyforagents.com/university/robots-txt-and-bot-rules)
-   [Which AI bots and agents to let in](https://amireadyforagents.com/decisions/which-bots-to-allow)

## Related terms

-   [User agent](https://amireadyforagents.com/glossary/user-agent)The name a browser or bot gives itself with each request.
-   [Content Signals](https://amireadyforagents.com/glossary/content-signals)A line in robots.txt that states how your content may be used, for example for search, AI input or training.
-   [Bot management](https://amireadyforagents.com/glossary/bot-management)Tools at your CDN, host or firewall that identify automated visitors and allow, challenge or block them.

## Sources

1.  [RFC 9309 Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (rfc-editor.org/rfc/rfc9309), read 4 Nov 2026.

Last reviewed 4 November 2026

Reviewed 4 November 2026. [1 source](#sources), each with the day it was read.