Six myths about AI agents and your website
In short
Most myths about AI agents treat all AI visitors as one thing, or a file or a score as a guarantee. robots.txt is a request, refusing training does not mean leaving search, llms.txt is optional, a scanner score is not a task test, a bot's name proves nothing, and most stores need no new payment method first.
Myth 1: robots.txt blocks AI bots
The standard behind robots.txt says its rules are not a form of access authorization. Well-behaved crawlers follow them; others may not. Blocking is also not what it seems: in Ora's lab, agents that were blocked still answered, from other sources, and missed more facts.
- 8 Oct 2026
Ora's lab found (8 Oct 2026) that in about 99 percent of journeys that hit a dead end on a site, the agent answered anyway from what it found elsewhere.
- 1 Sep 2022
RFC 9309 states that robots.txt rules are not a form of access authorization; enforcement needs bot management.
Do instead: make your bot protection enforce the position your robots.txt states.
Myth 2: refusing AI training means leaving AI search
Operators now separate the two, and most sites already do. A setting launched in September 2026 refuses training and keeps search.
- 30 Sep 2026
Cloudflare reported on 30 September 2026 that fewer than 1 percent of sites on its network block search crawlers, while 17 percent block training.
- 15 Sep 2026
Cloudflare launched a Disallow AI Training setting on 15 September 2026 that keeps accountable mixed-use crawlers from Apple, Google and Microsoft allowed for search while refusing training.
Do instead: decide training, search, assistants and agents as four separate choices.
Myth 3: every site needs an llms.txt file
It is optional. In Ora's experiment it changed nothing about whether an agent got the answer.
- 19 Oct 2026
Ora's 19-configuration experiment found llms.txt, JSON-LD, sitemaps and markdown mirrors did not change whether an agent retrieved the answer; only JavaScript-hidden answers and 403 blocks did.
Do instead: decide yes or not yet, and name who would keep the file current.
Myth 4: a high scanner score means agents will succeed
A scanner checks files, settings and page features. Only two changes stopped an agent in Ora's experiment, and both are a handful of checks among many.
- 19 Oct 2026
Ora's 19-configuration experiment (190 probes) found only two changes stopped retrieval: an answer hidden in JavaScript (0 of 5 for plain fetch) and a 403 to agent user agents (0 of 5 for both clients).
Do instead: read the failed checks that sit on your main task, then test the task itself.
Myth 5: a bot that says it is GPTBot is GPTBot
A name in a request is text anyone can copy. Verification closes the gap, and identification is now a commercial argument.
- 20 Sep 2026
Amazon said Muse did not identify itself while browsing its store, one of its reasons for blocking it in September 2026.
- Read 4 Nov 2026
Web Bot Auth is an IETF draft in which an agent signs its requests with HTTP Message Signatures so a site can check who operates it.
Do instead: ask your bot vendor whether it checks signatures, and see verifying agents.
Myth 6: being agent ready means adopting a new payment method
Agent payment routes exist, and none is universal. In the crawl very few sites publish a payment file at all.
- 19 Oct 2026
Ora's lab found agents paying through Stripe Link got 92.3 percent of test checkouts right (733 of 794 runs in a cloud browser), across 26 agent setups on the same 33 checkout tasks (AgentReady guide, October 2026).
Do instead: settle the price and fraud questions first. See agent payments.
Related
- The self-checkEighteen questions scored in your browser.
- Ask your web teamThe sentences to send, grouped by area.
Sources
- Ora research: AX is the new AEO (8 Oct 2026) (ora.ai/blog/ax-is-the-new-aeo), read 4 Nov 2026. Vendor claim.
- RFC 9309 Robots Exclusion Protocol (rfc-editor.org/rfc/rfc9309), read 4 Nov 2026.
- Cloudflare: The Internet has a second audience (30 Sep 2026) (blog.cloudflare.com/agentic-web), read 4 Nov 2026.
- Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026) (blog.cloudflare.com/accountable-mixed-use-ai-crawlers), read 4 Nov 2026.
- Introducing AgentReady.org (19 Oct 2026) (ora.ai/blog/agentready-launch), read 4 Nov 2026. Vendor claim.
- AgentReady.org guide (agentready.org), read 4 Nov 2026. Vendor claim.
- Retail Gazette: Amazon blocks Meta's Muse (22 Sep 2026) (retailgazette.co.uk/blog/2026/09/amazon-blocks-metas-new-ai-shopping-agent-from-its-website), read 4 Nov 2026. Reported by trade press.
- IETF draft: Web Bot Auth architecture (datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture), read 4 Nov 2026.
Last reviewed