What an AI assistant sees when it reads your site
A website can be perfectly readable to a person, score well in every SEO tool, and be very close to blank when an AI assistant fetches it. The reason is one technical detail that most site owners have never had to think about.
Two different documents at the same address
When you visit a page, your browser does two things. It downloads the HTML the server sends, and then it runs whatever JavaScript that HTML references, which may fetch more data and build most of the visible page in the browser itself. Modern frameworks lean heavily on the second step.
The result is that one URL has two versions. There is the source HTML, which is what arrives over the wire, and there is the rendered DOM, which is what exists after the JavaScript has run. On a client-rendered site these can be wildly different: the source may be a nearly empty shell with a script tag, while the rendered version is your whole article.
Google has run JavaScript for years, so search engines mostly see the second version. That is why this stopped being something anyone worried about. But the crawlers that feed AI assistants are a different population, and by and large they do not execute JavaScript. They fetch, they read what came back, and they move on. If your content only exists after rendering, it does not exist to them.
This is why a site can hold a good position in Google and be entirely absent from an assistant's answer about its own subject. The two systems are not looking at the same document.
Why the usual tools miss it
Most SEO auditors run in your browser or drive a headless one, so they inspect the rendered DOM by definition. They report your headings, your content depth and your structured data because all of it is present in the thing they are looking at. Every check passes. None of it says anything about what a crawler that does not render would find.
The only way to see the difference is to fetch the raw HTML the way a plain crawler would, and compare it against the rendered version. That comparison, how much of your content depends on JavaScript to exist, is the check AnswerRank is built around, and it is the one that most often explains an otherwise inexplicable result.
The bots to know about
AI crawlers identify themselves, and your robots.txt can allow or block them individually. It is worth knowing the names, because a blanket rule written years ago may be excluding you from systems that did not exist when it was written:
- GPTBot and OAI-SearchBot - OpenAI
- ClaudeBot - Anthropic
- PerplexityBot - Perplexity
- Google-Extended - Google's AI training and grounding
- Applebot-Extended - Apple
- CCBot - Common Crawl, which feeds many downstream datasets
There is a real decision here and it is not obvious. Blocking these keeps your work out of training data; it also keeps you out of the answers assistants give, including answers about you. Whichever you choose, it should be a choice rather than an inherited default.
What else makes a page usable to a machine
Once your content is actually in the source HTML, the remaining questions are about whether a model can tell what it says. In rough order of how much they matter:
- Answerability. An assistant is assembling a reply to a question. A page that states its answer plainly, early, in complete sentences, is far easier to quote than one that circles the point for six paragraphs before arriving. Front-loading is not dumbing down; it is putting the claim where it can be found.
- Structured data. Schema.org JSON-LD tells a machine what a page is rather than making it infer: an Article, its author, a date, a FAQ. Inference is where errors come from.
- Entity clarity. Say who you are and what you do in text, not only in a logo. A model that cannot establish what your organisation is has no reason to cite it.
- Heading structure. One H1, sensible H2s. Headings are how a long document gets chunked for retrieval, and bad structure means the wrong slice comes back.
- Author and freshness. A named author with a real page behind them, and a genuine date. Both feed the credibility judgement, and both are cheap to provide honestly.
- Semantic HTML. Article, section, nav, main. A page built entirely from divs is legible but structurally anonymous.
There is also llms.txt, a proposed convention: a markdown file at your root summarising the site and pointing at its most useful pages. Support is not universal and it may not become standard. It costs almost nothing to add.
The fix, when it applies
If the audit says your content is JavaScript-dependent, the remedy is to send it in the HTML. In practice that means server-side rendering or static generation. Every major framework supports it - Next.js, Nuxt, SvelteKit, Astro - and for a content site it is usually the default that was turned off rather than a rewrite.
You can check the raw version yourself without any tool. In a terminal:
curl -s https://yoursite.com | wc -w
If that returns a few dozen words for a page you know contains two thousand, you have found the problem. View Source, as distinct from Inspect Element, shows the same thing: Inspect shows the rendered DOM, View Source shows what actually arrived.
Why this is worth attention now
A growing share of questions get answered without anybody visiting a website. The assistant reads the sources and replies. Being one of the sources it can read is becoming a separate discipline from ranking in a results page, with different requirements and, at the moment, far less competition.
The unglamorous truth is that most of what helps is old advice that never stopped being right: send real HTML, structure it properly, say who wrote it, and answer the question you claim to answer. The novelty is only that a machine is now checking.