Home / Blog

SEO in the age of LLMs: be found by search engines and AI answers

More and more questions are answered by ChatGPT, Gemini, Perplexity and Google's AI Overviews before anyone clicks a link. Your website now has two audiences: search engines that rank pages, and AI systems that read them and decide what to quote.

By the Citewren team · 2 November 2026 · 8 min read

How people find information has changed

For twenty years, being discoverable meant ranking on a page of ten blue links. That page still exists, but an AI-written answer now often sits on top of it, and a growing share of questions skip the results page altogether and go straight to a chatbot.

The last number matters most for site owners. In Pew Research Center's analysis of real browsing data from 900 US adults in March 2025, people who saw an AI summary on Google clicked a traditional search result in 8% of visits, against 15% when no summary appeared. They clicked a link inside the summary itself in only 1% of visits. About 18% of the Google searches in the study produced an AI summary.

In other words: when an AI answer appears, fewer people click through, and the sources it names get most of what attention is left. Being one of those named sources is the new version of ranking on page one.

Classic SEO is still the foundation

It is tempting to treat AI search as a separate discipline with its own secret tricks. The official documentation says otherwise.

"There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary."
Google Search Central, AI features and your website

Google states that to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet. That is ordinary SEO: the page has to be crawlable, must not carry an accidental noindex, must return a 200 status, should have a sensible canonical URL, and must not block snippets with nosnippet.

The same logic applies outside Google. ChatGPT search, Perplexity and Claude can only cite pages their crawlers can fetch and read. A page that is broken for search engines (blocked, noindexed, empty without JavaScript, or impossible to understand) is almost always broken for AI engines too.

So the first rule of SEO for LLMs is simple: do not drop the basics. Titles, descriptions, headings, internal links, structured data and indexability still decide whether you are in the pool of pages any answer can draw from.

Where AI search is different

On top of that foundation, a few things matter more for AI engines than they did for classic search, and some of them are easy to get wrong without noticing.

1. AI engines use their own crawlers, and robots.txt treats them separately

Each AI company runs its own bots, and most now split them by purpose: one for building a search index, one for fetching a page when a user asks, and one for collecting model training data. Your robots.txt can allow or block each one individually.

This split gives you a real choice. Many sites decide to block training bots but allow search bots, so their content is not used to train models but can still be found and cited. The mistake to avoid is a blanket rule written for "AI bots" that also blocks the search crawlers you want.

It can also happen without anyone on your team touching robots.txt. In July 2025 Cloudflare, which sits in front of a large share of the web, switched to blocking AI crawlers by default for sites it hosts, with the option to allow them per bot. If your site is behind Cloudflare or another CDN or firewall, check its bot settings as well as your robots.txt. We did exactly that for citewren.com.

2. Many AI crawlers do not run JavaScript

Googlebot renders JavaScript before indexing. Most AI crawlers do not. In an analysis of crawler traffic across its network published in December 2024, Vercel found that none of the major AI crawlers it observed, including OpenAI's crawlers and ClaudeBot, rendered JavaScript. They read the HTML the server sends and nothing more.

If your product descriptions, prices, articles or FAQs are injected by client-side JavaScript, an AI crawler may see an empty shell. Server-side rendering or static HTML for your important content fixes this. A quick test is to view the page source (not the inspector) and check whether the text you care about is there.

3. Being quotable matters as much as being found

An AI engine does not show your page; it reads it, extracts what it needs and writes its own answer. Content that is easy to extract and safe to repeat has an advantage.

The research backs this up. In GEO: Generative Engine Optimization (Aggarwal et al., KDD 2024), the researchers tested how different rewrites change a page's visibility in generative engine answers. Adding relevant statistics, quotations from credible sources, and citations to sources improved visibility by roughly 30 to 40% on their main metric. Keyword stuffing, the classic SEO shortcut, did not help and in some tests performed worse than doing nothing.

In practice, quotable content tends to:

• answer the question directly in the first sentence or two of a section, then add detail;
• use headings that read like the questions people actually ask;
• put comparisons and specifications in lists and tables rather than long paragraphs;
• include concrete numbers and name their sources;
• show who wrote it and when it was last updated, with clear About and Contact pages.

None of this is new advice for good web writing. AI engines simply reward it more directly.

4. The success metric shifts from rankings to mentions

In classic search you can check your position for a keyword. In AI answers there is no fixed position: the same question can produce a different answer each time, and the answer may mention your brand without linking to you. Measuring AI visibility means asking the questions your customers ask, repeatedly, and tracking how often you are mentioned or cited compared with competitors. This is what people mean by generative engine optimization, or GEO.

What about llms.txt?

llms.txt is a proposed standard, published in September 2024, for a Markdown file at the root of a site that gives language models a short guide to its most useful pages. It is cheap to add and can help tools and agents that look for it. But keep expectations realistic: Google says you do not need to create "new machine readable files, AI text files, or markup" to appear in its AI features, and we are not aware of any major AI search engine that has said publicly that it uses llms.txt to decide what to cite. Treat it as a nice extra, not a replacement for the steps above.

A checklist for being discoverable by both

1. Indexable: the page returns 200, has no stray noindex or X-Robots-Tag, and has a correct canonical URL.
2. Snippet-eligible: no nosnippet or overly tight max-snippet on content you want quoted.
3. AI search crawlers allowed: Googlebot, OAI-SearchBot, Claude-SearchBot and PerplexityBot are not blocked in robots.txt, or at your CDN or firewall. Decide separately about training bots such as GPTBot, ClaudeBot and Google-Extended.
4. Visible without JavaScript: the main content is in the server-rendered HTML.
5. Clear structure: one H1, logical headings, question-style subheadings, answer-first paragraphs, lists and tables.
6. Evidence: specific facts and numbers, with named sources.
7. Trust signals: author, publication and update dates, About and Contact pages.
8. Structured data: valid JSON-LD for the page type (Article, Product, FAQPage, Organization).
9. Measure: check whether AI Overviews and AI assistants cite you for the questions that matter to your business, and recheck over time.

The bottom line

SEO has not been replaced. It has gained a second audience. The fundamentals that get a page indexed and ranked are the same ones that make it eligible for AI answers. What has changed is the cost of small mistakes: a robots rule, a CDN default or a JavaScript-only page can make you invisible to AI engines while your Google rankings look fine. And because fewer people click through when an AI answer appears, being the source that answer names is increasingly where visibility is won.

Sources

1. Google Search Central, AI features and your website (last updated 10 December 2025).
2. Google Search Central, Google's common crawlers (Google-Extended).
3. Pew Research Center, Google users are less likely to click on links when an AI summary appears in the results, 22 July 2025.
4. TechCrunch, Google's AI Overviews have 2B monthly users, 23 July 2025.
5. TechCrunch, Sam Altman says ChatGPT has hit 800M weekly active users, 6 October 2025.
6. OpenAI, Overview of OpenAI crawlers.
7. Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
8. Perplexity, Perplexity crawlers.
9. MIT Technology Review, Cloudflare will now, by default, block AI bots from crawling its clients' websites, 1 July 2025.
10. Vercel, The rise of the AI crawler, 17 December 2024.
11. Aggarwal et al., GEO: Generative Engine Optimization, KDD 2024.
12. Jeremy Howard, The /llms.txt file, September 2024.