--- title: "Six myths about AI agents and your site | Am I Ready for Agents?" description: "Six common myths about AI agents and your website, each checked against dated, sourced facts. What to do instead, one line each." canonical: https://amireadyforagents.com/briefings/six-myths-about-ai-agents-and-your-website site: "Am I Ready for Agents?" last_updated: 2026-11-04 section: "Guides" --- # Six myths about AI agents and your site | Am I Ready for Agents? > Six common myths about AI agents and your website, each checked against dated, sourced facts. What to do instead, one line each. Source: https://amireadyforagents.com/briefings/six-myths-about-ai-agents-and-your-website Site: Am I Ready for Agents? Last reviewed: 2026-11-04 # Six myths about AI agents and your website ## In short Most myths about AI agents treat all AI visitors as one thing, or a file or a score as a guarantee. robots.txt is a request, refusing training does not mean leaving search, llms.txt is optional, a scanner score is not a task test, a bot's name proves nothing, and most stores need no new payment method first. ## Myth 1: robots.txt blocks AI bots The standard behind robots.txt says its rules are not a form of access authorization. Well-behaved crawlers follow them; others may not. Blocking is also not what it seems: in Ora's lab, agents that were blocked still answered, from other sources, and missed more facts. - 8 Oct 2026 Ora's lab found (8 Oct 2026) that in about 99 percent of journeys that hit a dead end on a site, the agent answered anyway from what it found elsewhere. Vendor claim. [Ora research: AX is the new AEO (8 Oct 2026)](https://ora.ai/blog/ax-is-the-new-aeo?utm_source=amireadyforagents.com&utm_medium=referral&utm_campaign=briefing), read 4 Nov 2026 - 1 Sep 2022 RFC 9309 states that robots.txt rules are not a form of access authorization; enforcement needs bot management. [RFC 9309 Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309), read 4 Nov 2026 Do instead: make your bot protection enforce the position your robots.txt states. ## Myth 2: refusing AI training means leaving AI search Operators now separate the two, and most sites already do. A setting launched in September 2026 refuses training and keeps search. - 30 Sep 2026 Cloudflare reported on 30 September 2026 that fewer than 1 percent of sites on its network block search crawlers, while 17 percent block training. [Cloudflare: The Internet has a second audience (30 Sep 2026)](https://blog.cloudflare.com/agentic-web/), read 4 Nov 2026 - 15 Sep 2026 Cloudflare launched a Disallow AI Training setting on 15 September 2026 that keeps accountable mixed-use crawlers from Apple, Google and Microsoft allowed for search while refusing training. [Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/), read 4 Nov 2026 Do instead: decide training, search, assistants and agents as four separate choices. ## Myth 3: every site needs an llms.txt file It is optional. In Ora's experiment it changed nothing about whether an agent got the answer. - 19 Oct 2026 Ora's 19-configuration experiment found llms.txt, JSON-LD, sitemaps and markdown mirrors did not change whether an agent retrieved the answer; only JavaScript-hidden answers and 403 blocks did. Vendor claim. [Introducing AgentReady.org (19 Oct 2026)](https://ora.ai/blog/agentready-launch?utm_source=amireadyforagents.com&utm_medium=referral&utm_campaign=briefing), read 4 Nov 2026 Do instead: decide yes or not yet, and name who would keep the file current. ## Myth 4: a high scanner score means agents will succeed A scanner checks files, settings and page features. Only two changes stopped an agent in Ora's experiment, and both are a handful of checks among many. - 19 Oct 2026 Ora's 19-configuration experiment (190 probes) found only two changes stopped retrieval: an answer hidden in JavaScript (0 of 5 for plain fetch) and a 403 to agent user agents (0 of 5 for both clients). Vendor claim. [AgentReady.org guide](https://agentready.org/?utm_source=amireadyforagents.com&utm_medium=referral&utm_campaign=briefing), read 4 Nov 2026 Do instead: read the failed checks that sit on your main task, then test the task itself. ## Myth 5: a bot that says it is GPTBot is GPTBot A name in a request is text anyone can copy. Verification closes the gap, and identification is now a commercial argument. - 20 Sep 2026 Amazon said Muse did not identify itself while browsing its store, one of its reasons for blocking it in September 2026. Trade press report. [Retail Gazette: Amazon blocks Meta's Muse (22 Sep 2026)](https://www.retailgazette.co.uk/blog/2026/09/amazon-blocks-metas-new-ai-shopping-agent-from-its-website/), read 4 Nov 2026 - Read 4 Nov 2026 Web Bot Auth is an IETF draft in which an agent signs its requests with HTTP Message Signatures so a site can check who operates it. [IETF draft: Web Bot Auth architecture](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/), read 4 Nov 2026 Do instead: ask your bot vendor whether it checks signatures, and see [verifying agents](https://amireadyforagents.com/decisions/verifying-agents). ## Myth 6: being agent ready means adopting a new payment method Agent payment routes exist, and none is universal. In the crawl very few sites publish a payment file at all. - 19 Oct 2026 Ora's lab found agents paying through Stripe Link got 92.3 percent of test checkouts right (733 of 794 runs in a cloud browser), across 26 agent setups on the same 33 checkout tasks (AgentReady guide, October 2026). Vendor claim. [AgentReady.org guide](https://agentready.org/?utm_source=amireadyforagents.com&utm_medium=referral&utm_campaign=briefing), read 4 Nov 2026 Do instead: settle the price and fraud questions first. See [agent payments](https://amireadyforagents.com/decisions/agent-payments). ## Related - [The self-check](https://amireadyforagents.com/self-check)Eighteen questions scored in your browser. - [Ask your web team](https://amireadyforagents.com/ask-your-web-team)The sentences to send, grouped by area. ## Sources 1. [Ora research: AX is the new AEO (8 Oct 2026)](https://ora.ai/blog/ax-is-the-new-aeo?utm_source=amireadyforagents.com&utm_medium=referral&utm_campaign=briefing) (ora.ai/blog/ax-is-the-new-aeo), read 4 Nov 2026. Vendor claim. 2. [RFC 9309 Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) (rfc-editor.org/rfc/rfc9309), read 4 Nov 2026. 3. [Cloudflare: The Internet has a second audience (30 Sep 2026)](https://blog.cloudflare.com/agentic-web/) (blog.cloudflare.com/agentic-web), read 4 Nov 2026. 4. [Cloudflare: Disallow AI Training, Accountable crawlers (15 Sep 2026)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/) (blog.cloudflare.com/accountable-mixed-use-ai-crawlers), read 4 Nov 2026. 5. [Introducing AgentReady.org (19 Oct 2026)](https://ora.ai/blog/agentready-launch?utm_source=amireadyforagents.com&utm_medium=referral&utm_campaign=briefing) (ora.ai/blog/agentready-launch), read 4 Nov 2026. Vendor claim. 6. [AgentReady.org guide](https://agentready.org/?utm_source=amireadyforagents.com&utm_medium=referral&utm_campaign=briefing) (agentready.org), read 4 Nov 2026. Vendor claim. 7. [Retail Gazette: Amazon blocks Meta's Muse (22 Sep 2026)](https://www.retailgazette.co.uk/blog/2026/09/amazon-blocks-metas-new-ai-shopping-agent-from-its-website/) (retailgazette.co.uk/blog/2026/09/amazon-blocks-metas-new-ai-shopping-agent-from-its-website), read 4 Nov 2026. Reported by trade press. 8. [IETF draft: Web Bot Auth architecture](https://datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture/) (datatracker.ietf.org/doc/draft-meunier-web-bot-auth-architecture), read 4 Nov 2026. Last reviewed 4 November 2026 Reviewed 4 November 2026. [8 sources](#sources), each with the day it was read.