🤖 AI & Agents · 9 min read
Is Your Website Agent-Ready? How to Check It Yourself
A site is agent-ready when four machine-readable things are published and reachable: robots.txt rules that name AI crawlers, an llms.txt reading list, an agent-tools.json task list, and WebMCP tool declarations on the pages that do real work. All seven checks below run from a terminal in about five minutes.
The reason to check rather than assume is that every one of these layers fails quietly. A file can return 200 and still be the wrong file. A declaration can list an action that no page can perform. A sitemap can describe a hostname you are not testing. We run these checks on our own sites before we run them on anyone else's, which is where the numbers further down come from.
What each layer actually answers
| File | The question it answers | Who reads it |
|---|---|---|
| robots.txt | May I fetch this page? | Crawlers, including AI crawlers |
| llms.txt | What here is worth opening? | Assistants assembling an answer |
| agent-tools.json | What can this site actually do? | Assistants choosing an action |
| WebMCP declarations | How do I do it on this page? | Browser agents with the trial enabled |
The four are not rivals. Access, reading list, task list, execution. A site can publish three of them and still be useful to an assistant, which is what two of the three sites in our own test do.
The seven checks, in the order we run them
- Does robots.txt name AI crawlers?
curl -s https://example.com/robots.txt | head -40. Look for groups that name GPTBot, ClaudeBot, PerplexityBot and similar. One trap is worth knowing before you conclude anything: consecutiveUser-agentlines form a single group and share whatever rules follow, so a file with sixteen agent names may carry one rule set rather than sixteen. We walked through the AI crawler tokens separately. - Does llms.txt come back as text? Print the type and the size, not just the status code:
You wantcurl -s -o /dev/null -w "http=%{http_code} type=%{content_type} bytes=%{size_download}\n" \ https://example.com/llms.txttext/plainand a byte count that looks like a real file. What to put in it is a separate article. - Does agent-tools.json return JSON? The same command against
/agent-tools.json. This is the check that catches a published file that is not published at all, and we have been burned by it once already. The field-by-field guide explains what belongs in the file. - Does the sitemap list the host you intend to keep?
curl -s https://example.com/sitemap.xml | grep -o "<loc>[^<]*" | head -5. Two of the three sites we audited answer on a preview hostname while the sitemap names the production domain, which is usually deliberate and always worth confirming. - Do the controls an assistant needs have stable names? A file input with a label, a submit button with a label, no field that is generated by JavaScript after a click. An agent that cannot name an element cannot target it.
- Is the WebMCP origin-trial header present?
curl -sI https://example.com | grep -i origin-trial. Nothing here works without the token, and the token carries an expiry date you should write down. - Does a browser agree? Open a Chromium build with
chrome://flags/#enable-webmcp-testingenabled and readawait navigator.modelContext.getTools(), or use the Model Context Tool Inspector extension Chrome documents. This is the only check that proves the declaration reached an agent. The mechanics are in What Is WebMCP?
What we measured on three live sites today
All three sites are ours, so the failures are ours to publish. The audit ran on 21 September 2026 and the numbers below are its raw output, not a summary.
| Check | nocodecsv.com | carecosttool.pages.dev | Second site we run |
|---|---|---|---|
| agent-tools.json | 5 tools, valid JSON | 2 tools, valid JSON | 6 tools, valid JSON |
| llms.txt | 10,980 characters, has a For AI agents section | 2,783 characters, has the section | 1,436 characters, no such section |
| AI crawlers named in robots.txt | 14 | 13 | 0 |
| sitemap.xml | 81 URLs, host matches | 7 URLs, different host | 11 URLs, different host |
| WebMCP origin-trial header | present, 196 characters | none | none |
| Browser getTools() result | 3 tools listed | not enabled | not enabled |
One site out of three has the full stack. The other two publish the static files, which is the part that depends only on them, and neither has the in-page layer, because the origin-trial token depends on a Chrome programme rather than on the site owner. That split is the useful finding: the readable half of agent-readiness is a decision you can make today, and the operable half has a waiting list.
Three results that changed how we check
1. A 200 is not proof the file exists. On a static host that falls back to a single-page app shell, an unknown path returns 200 with content type text/html and tens of kilobytes of HTML. A monitoring check that watches for 404 will never notice. Print the type, as in check two and three above. We caught a real instance of this on 19 September and confirmed it had cleared a day later; the status code was 200 on both days.
2. A sitemap can name a host you are not testing. Two of the three sites answer on a preview hostname while the sitemap lists a custom domain. That is normally intentional, and it also means the host you audit should be the host you intend to ship, because the sitemap is what a crawler follows.
3. A published agent-tools.json is not a crawler policy. The second site lists six actions and names zero AI crawlers in robots.txt. RFC 9309 makes the default allow, so nothing is blocked, but nothing invites the crawlers either. Naming them is the only way a site states a position, and the absence is what the audit reports.
Run the same audit on your own site
The script behind that table is open source at github.com/CindyLiao1106/agent-ready-audit, and it is the same one we run on client work:
python3 agent_ready_audit.py https://example.com --md report.mdIt fetches agent-tools.json, llms.txt, robots.txt and sitemap.xml, counts the AI crawler names, checks the origin-trial header, probes the pages for declared tools, and where a browser build is available it reads the tool list from getTools(). It reports what it could not test as plainly as what it found. On our calculator project it prints "not enabled" for the browser check rather than a green tick, because no token is present, and that is the correct answer.
The order that worked for us
- robots.txt first, because if crawlers are not allowed in, nothing else matters.
- llms.txt second, because it is prose and takes the least thought.
- agent-tools.json third, verifying every entry against a live page as you write it.
- WebMCP declarations last, page by page, because that work scales with the number of actions rather than the size of the site.
- A re-check on a date. Our trial token expires 17 November 2026. Origin trials end, sitemaps drift, and a declaration file stops matching the pages under it the moment someone renames a field.
Frequently asked questions
What does agent-ready mean?
A site is agent-ready when four machine-readable things are published and reachable: robots.txt rules that name AI crawlers, an llms.txt reading list, an agent-tools.json task list, and WebMCP tool declarations on the pages that do real work. Without them an assistant can only guess what your site does from page text.
Is agent-ready an official standard?
Partly. robots.txt follows RFC 9309, which is a published standard. llms.txt is a community convention documented at llmstxt.org and is not enforced by anything. WebMCP is a Chrome API in origin trial. agent-tools.json has no specification at all; it is a naming convention we and other sites have adopted, so treat it as a convention rather than a rule.
How do I check whether my website is agent-ready?
Run seven checks in this order: confirm robots.txt names AI crawlers, confirm llms.txt returns plain text, confirm agent-tools.json returns JSON instead of HTML, confirm the sitemap lists the host you intend to keep, confirm your key form controls have stable labels, confirm the WebMCP origin-trial header is present, and finally confirm in a browser that getTools() lists your tools.
Do I need WebMCP to be agent-ready?
No. The three static files make your site readable and citable without any browser support. WebMCP is what lets an assistant act on a page rather than describe it, so a site can be agent-readable today and add the action layer later. In our own three-site measurement, two sites published the static files and only one had the in-page layer.
Will being agent-ready make ChatGPT cite my site?
No one can promise that, including us. These files remove technical blockers: a crawler that is allowed in, a reading list, a task list and declared tools. Whether a particular assistant retrieves and quotes your page is that assistant's decision, based on its own index and policy.
How long does the work take?
The two text files are an afternoon, and the agent-tools.json that follows them is another hour if you verify each entry against the live page. Tool declarations are per page, so that part scales with how many actions your site actually offers. Our own file lists five actions across four pages.
What is the fastest way to see where my site stands?
Run a single command against your domain: three fetches that check for agent-tools.json, llms.txt and the AI crawler lines in robots.txt. The open-source script we use, agent_ready_audit.py, does those three plus the sitemap, the origin-trial header and a browser check, and writes a Markdown report.
Can an assistant fill in my contact form once the declarations are in place?
For text, dropdown and number fields, yes, and we have measured a declared form being driven end to end. For file inputs, no: in our test a file picker did not become a tool parameter, so the tool was discoverable but its parameter list came back empty. Any workflow that starts with an upload is still out of reach for a browser agent.
Tools mentioned in this guide
Three tools that fit the work described above, and why each one is on the list:
- OpenCode Go — the audit is a script, and the fixes after it are code edits against your own markup. That loop is faster with an assistant in the terminal. Try OpenCode Go
- Stack AI — if an agent action should land somewhere you can read later, a workflow can turn an incoming request into a record instead of a page view you never see. Try Stack AI
- Softr — check four asks whether your sitemap names the host you intend to keep. If your catalogue lives in a spreadsheet, publishing it as a no-code app gives every item a stable URL to point at. Try Softr
Some links above are affiliate links — if you buy through them we may earn a commission at no extra cost to you. OpenCode Go uses our referral link; the other two currently point to each vendor's official page until our tracking links are approved.
Want Your Site Checked and Then Fixed?
Send a URL and you get the same findings note the script above writes, read by a person and put in priority order. Implementation is quoted as a fixed scope afterwards.
I build this layer for other sites — llms.txt, agent-tools.json, WebMCP declarations — and verify each one against the live pages. Details at /agent-ready.
Related reading
AI & Analysis — other guides that pair well with this one.
Browse all guides in the NoCodeCSV blog.