The AI Agent Revolution Is Already Here
AI agents are no longer a future concept — they are actively browsing the web right now on behalf of millions of users. When someone asks Claude, ChatGPT, or Perplexity a question about anything, these agents don't just show search results. They visit websites, read content, check APIs, and synthesise answers directly.
The question is: is your website ready to be understood by an AI agent?
Most websites aren't. They were built for humans — with rich visuals, navigation menus, and marketing copy. AI agents need something different: structured metadata, machine-readable endpoints, and programmatic discovery signals.
We recently went through the process of making venzotechnologies.com fully agent-ready, achieving a 100% score on isitagentready.com.
Here is exactly what we did, and how you can do the same for any website.
What Does Agent-Ready Actually Mean?
An agent-ready website allows AI agents to:
- Discover what the site offers without parsing HTML
- Understand the authentication scheme before making API calls
- Fetch structured, token-efficient summaries of content
- Find machine-readable service descriptions and capabilities
The isitagentready.com tool tests for 12 checks across 4 categories:
| Category | Checks | What It Tests |
|---|---|---|
| Discoverability | 3 | llms.txt, structured data, link discovery |
| Content | 1 | JSON-LD schemas, WebMCP tools |
| Bot Access Control | 2 | robots.txt, Content-Signal directive |
| API, Auth, MCP & Skill Discovery | 6 | OAuth metadata, API catalog, MCP server card, agent skills |
The 13 Things We Implemented
1. /llms.txt — The LLM Context File
Think of this as robots.txt but for language models. Instead of permission rules, it's a plain-text, token-efficient summary of your entire website — company overview, services, navigation structure, contact details — all in a format LLMs can ingest efficiently.
Why it matters: When an AI agent visits your site, it can fetch /llms.txt first and immediately understand everything about your business without parsing hundreds of HTML pages.
2. Updated robots.txt with AI Crawlers
Modern AI assistants use their own crawlers. We explicitly allowed: - GPTBot, ChatGPT-User (OpenAI) - Claude-Web, anthropic-ai (Anthropic) - PerplexityBot, Google-Extended, Copilot, DuckAssistBot and more
We also added the Content-Signal directive — a new standard that lets you declare AI usage permissions in a single line:
Content-Signal: ai-train=no, search=yes, ai-input=yes
This tells AI companies: "Don't train on our content, but you can search-index it and use it as context for user queries."
3. Rich JSON-LD Structured Data
We expanded the site's JSON-LD schemas to include:
- Organization schema with knowsAbout fields listing our domain expertise
- WebSite schema with a SearchAction so agents know how to search the site programmatically
- Proper @id references linking the two schemas together
4. HTTP Link Response Headers (RFC 8288)
Every single HTTP response from the site now includes a Link header that points agents to all discovery resources:
Link: </.well-known/api-catalog>; rel="api-catalog",
</.well-known/api-catalog>; rel="service-desc"; type="application/linkset+json",
</llms.txt>; rel="service-doc"; type="text/plain",
</llms.txt>; rel="describedby"; type="text/plain",
</.well-known/mcp/server-card.json>; rel="https://modelcontextprotocol.io/server-card",
</.well-known/agent-skills/index.json>; rel="https://agentskills.io/discovery"
This means an agent making any request to our site — even without knowing our URL structure — immediately learns about all our machine-readable resources.
5. Markdown-for-Agents Content Negotiation
When an AI agent sends a request with Accept: text/markdown, our server returns a pre-built markdown version of the page instead of HTML. This is more token-efficient for LLMs and doesn't require them to parse HTML, strip scripts, and extract content.
6. API Catalog (RFC 9727)
The /.well-known/api-catalog endpoint (served as application/linkset+json) is a machine-readable catalog of all the APIs and documents our site exposes. It follows RFC 9727 — a formal IETF standard for API discovery.
7–9. OAuth & Authentication Metadata
Even though our site doesn't have a live OAuth server yet, publishing authentication metadata stubs is important because:
- Agents need to know the auth scheme before attempting API calls
- It signals to agents that the site is aware of programmatic access patterns
- Future-proofs the site when real APIs are added
We published three endpoints: - /.well-known/openid-configuration — OIDC Discovery (for identity) - /.well-known/oauth-authorization-server — OAuth 2.0 Server Metadata (RFC 8414) - /.well-known/oauth-protected-resource — Protected Resource Metadata (RFC 9728)
10. MCP Server Card
The Model Context Protocol (MCP) Server Card at /.well-known/mcp/server-card.json is a structured JSON document that tells MCP-compatible AI clients (like Claude Desktop) exactly what resources and tools our server exposes, what authentication is required, and where the MCP endpoint lives.
11. Agent Skills Discovery Index
The /.well-known/agent-skills/index.json file follows the Agent Skills Discovery RFC v0.2.0. It's a catalogue of all "skills" the site exposes — our API catalog, MCP server card, and llms.txt — each with a SHA-256 integrity digest for verification.
12. WebMCP Browser Tools
We added a navigator.modelContext.provideContext() script to every page. This W3C WebMCP API lets AI agents running in a browser context discover and call tools directly on the page — such as get_company_info, list_services, and navigate_to.
The Tricky Part: The .well-known Dot-Directory Problem
This is where most developers get stuck.
The problem: Astro (and most other frameworks) silently ignore directories starting with a dot (.) when routing. So src/pages/.well-known/api-catalog.ts simply doesn't work — Astro never registers it as a route.
Static files in public/.well-known/ are served, but without SSR you can't control the Content-Type header — and /.well-known/api-catalog must be served as application/linkset+json, not text/plain or application/octet-stream.
Our fix: We used Astro's injectRoute() API inside a custom integration to programmatically register these routes, bypassing the dot-directory restriction entirely:
// astro.config.mjs
const wellKnownRoutes = {
name: 'well-known-routes',
hooks: {
'astro:config:setup': ({ injectRoute }) => {
injectRoute({ pattern: '/.well-known/api-catalog', entrypoint: 'src/well-known/api-catalog.ts', prerender: false });
injectRoute({ pattern: '/.well-known/openid-configuration', entrypoint: 'src/well-known/openid-configuration.ts', prerender: false });
injectRoute({ pattern: '/.well-known/oauth-authorization-server', entrypoint: 'src/well-known/oauth-authorization-server.ts', prerender: false });
injectRoute({ pattern: '/.well-known/oauth-protected-resource', entrypoint: 'src/well-known/oauth-protected-resource.ts', prerender: false });
},
},
};
Each handler is a simple Astro API route that returns JSON with the correct Content-Type.
The Result: 100 / 100
After implementing all 13 features:
- Discoverability: 3/3
- Content: 1/1
- Bot Access Control: 2/2
- API, Auth, MCP & Skill Discovery: 6/6
- Total: 100 — Level 6
Should You Do This for Your Website?
If your website represents a business that wants to be found, understood, and recommended by AI assistants — yes, absolutely.
The investment is relatively small (a few hours of engineering work), and the payoff grows every month as more users rely on AI agents to research, compare, and recommend services.
The web is changing. Sites that are agent-ready will be cited, summarised, and recommended by AI tools. Sites that aren't will be invisible to the fastest-growing category of web traffic.
