In this article

For more than two decades, getting found online meant ranking in a list of ten blue links, and the whole SEO playbook was built around earning a click from that list. On a growing share of searches, that list is now the second thing people see, below an answer a language model wrote.
The numbers moved fast. When Pew Research tracked the real Google searches of 900 US adults in March 2025, people clicked a traditional result on 8% of visits when an AI summary appeared, against 15% when one didn’t. By December 2025, Ahrefs measured a 58% lower click-through rate for the top-ranking page on queries with an AI Overview. Google says AI Mode passed one billion monthly users in May 2026, a year after launch.
This post covers what AEO and GEO mean, how AI answers pick sources, and what to change, with scripts I ran against real sites.
Summary
- Clicks fall where AI answers appear: an 8% vs 15% click rate in Pew’s data, and a 58% lower position 1 CTR in Ahrefs’ study.
- AEO and GEO are still SEO, says Google’s own guide: pages must be indexed and snippet-eligible, and Google Search ignores
llms.txt. - Citations come from fan-out queries. Only 37.9% of AI Overview citations ranked in the top 10 in Ahrefs’ March 2026 sample.
- Let the search bots in and serve the answer as HTML, because the major AI crawlers don’t run JavaScript.
- Track your AI mentions with Lumirank, which runs your prompts daily on ChatGPT, Gemini, Perplexity, Copilot, AI Mode and AI Overviews, then back it up with Search Console’s Generative AI report.
What the click data shows
Pew’s study covered 68,879 searches, 18% of which produced an AI summary. On those pages, users clicked a link inside the summary on just 1% of visits. Ahrefs’ 300,000-keyword comparison, charted below, has position 1 CTR falling from 7.3% to 1.6% on AI Overview queries. SparkToro, using Similarweb’s panel, counted 68.01% of Google searches ending without a click in January to April 2026.
Two things complicate this. Being cited cushions the loss: Seer Interactive's 2026 study of 53 brands found cited pages earned 120% more organic clicks per impression than uncited ones, though 38% fewer than on queries with no AI Overview. And Google disputes the decline: Liz Reid wrote in August 2025 that organic click volume was “relatively stable year-over-year”, and Alphabet said in April 2026 that queries were at an all-time high. Both can be true: more searches, fewer clicks per search.

Data: Ahrefs, 2026.
SEO, AEO and GEO: three names for one pipeline
AEO, answer engine optimization, dates from at least 2018, the featured-snippet and voice-assistant era, and meant getting picked as the single boxed or spoken answer. GEO comes from a November 2023 paper by Pranjal Aggarwal and co-authors, accepted at KDD 2024, which measured how page edits change visibility inside LLM-generated answers. Today the terms are used interchangeably.
Google’s generative AI optimization guide, published May 15, 2026, defines both and then says: “From Google Search’s perspective, optimizing for generative AI search is optimizing for the search experience, and thus still SEO.”
The difference is the output. SEO competes for a position in a list and counts clicks. AEO and GEO compete to be one of the few sources a model cites, and count citations and mentions. The machinery underneath is the same: crawl -> index -> retrieve -> rank, with a language model at the end.
How an AI answer picks the pages it cites
For AI Overviews and AI Mode, the model generates a query fan-out, which Google defines as “a set of concurrent, related queries generated by the model to request more information and fetch additional relevant search results.” Google’s core ranking systems retrieve pages for those queries from the Search index, and the model grounds its answer in them. A cited page must be indexed and eligible to show with a snippet, and since August 31, 2026 the site must not be set to Exclude in Search Console’s Search generative AI control (the default is Include).
Fan-out is why citations drift away from the classic ranking. Take this post’s worked example, a docs page titled “How to rotate an API key without downtime.” The question might fan out into “API key rotation grace period” or “revoke an old API key safely” (my illustrations; Google doesn’t publish the fan-out). A page that ranks for any sub-query can be cited without ranking for the main question. Ahrefs found 76.1% of AI Overview citations ranked in the top 10 in July 2025, and only 37.9% in a March 2026 sample of 4 million URLs. Its parsing changed between studies, so trust the direction more than the size.
Other engines run the same loop on other indexes. ChatGPT search “rewrites your query into one or more targeted queries” for its search providers, which OpenAI’s Enterprise docs name as Bing. Anthropic’s subprocessor list names Brave Search for web search, and Copilot is grounded in Bing, so verify your site in Bing Webmaster Tools too.

The sub-queries are illustrative.
Step 1: Let the AI search bots in
AI companies split crawling across separate user-agent tokens. OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages for search answers. GPTBot and ClaudeBot collect training data, and Google-Extended covers Gemini training and grounding, not Search inclusion. Blocking a training bot doesn’t remove you from AI search; blocking a search bot does (OpenAI says ChatGPT search picks up a robots.txt change in about 24 hours). AI Overviews have no separate bot: Googlebot’s rules apply.
Check a site with Python’s standard-library parser, one token at a time. Save this as ai_bots.py:
import sys
import urllib.request
import urllib.robotparser
from urllib.parse import urljoin
BOTS = {
"Googlebot": "Google Search, AI Overviews, AI Mode",
"Bingbot": "Bing, Copilot",
"OAI-SearchBot": "ChatGPT search results",
"ChatGPT-User": "ChatGPT fetching a page for a user",
"Claude-SearchBot": "Claude search results",
"PerplexityBot": "Perplexity search results",
"GPTBot": "OpenAI model training",
"ClaudeBot": "Anthropic model training",
"Google-Extended": "Gemini training and grounding",
}
url = sys.argv[1]
req = urllib.request.Request(urljoin(url, "/robots.txt"),
headers={"User-Agent": "Mozilla/5.0"})
with urllib.request.urlopen(req, timeout=10) as resp:
lines = resp.read().decode("utf-8", "replace").splitlines()
rp = urllib.robotparser.RobotFileParser()
rp.parse(lines)
for bot, role in BOTS.items():
verdict = "allowed" if rp.can_fetch(bot, url) else "BLOCKED"
print(f"{bot:<17} {verdict:<8} {role}")Output against a major news site, run on September 29, 2026 with Python 3.13:
$ python3 ai_bots.py https://www.nytimes.com/section/technology
Googlebot allowed Google Search, AI Overviews, AI Mode
Bingbot allowed Bing, Copilot
OAI-SearchBot BLOCKED ChatGPT search results
ChatGPT-User BLOCKED ChatGPT fetching a page for a user
Claude-SearchBot BLOCKED Claude search results
PerplexityBot BLOCKED Perplexity search results
GPTBot BLOCKED OpenAI model training
ClaudeBot BLOCKED Anthropic model training
Google-Extended BLOCKED Gemini training and groundingThe New York Times keeps Google and Bing open and blocks every AI bot, deliberately. If yours does that by accident, look for a CDN “block AI bots” toggle. Python’s parser takes the first matching rule, not the longest as RFC 9309 requires, so check overlapping Allow and Disallow paths with Google’s open-source robotstxt parser. User-triggered fetchers play looser: OpenAI says robots.txt “may not apply” to ChatGPT-User.
Step 2: Put the answer in the HTML
Vercel and MERJ’s crawler study found “none of the major AI crawlers currently render JavaScript”, including OAI-SearchBot, ChatGPT-User, GPTBot, ClaudeBot and PerplexityBot. Gemini does render, through Googlebot’s infrastructure. So a client-rendered page can appear in AI Overviews and look blank to ChatGPT. The study is from December 2024; I found no newer one.
Count the words a non-rendering crawler receives:
curl -s -A "OAI-SearchBot/1.3" https://example.com/docs/rotate-api-keys/ \
| python3 -c "import sys,re,html; t=re.sub(r'(?s)<(script|style)\b.*?</\1>|<[^>]+>',' ',sys.stdin.read()); print(len(html.unescape(t).split()),'words in the raw HTML')"Against a static Hugo post on this blog it printed 2552 words in the raw HTML. Against excalidraw.com, a client-rendered app, it printed 12: the whole body read “You need to enable JavaScript to run this app.” Fine for a whiteboard, but a docs page needs the answer in the server response: server-side rendering, static generation or prerendering.
Step 3: Write passages a model can lift
The GEO paper tested which edits move citations in a controlled setup. On its 10,000-query benchmark, adding quotations, statistics and cited sources raised visibility in generated answers by 30-40% on its position-adjusted word count metric. Keyword stuffing scored below the unedited baseline (17.7 against 19.3). Treat that as directional: the test engine was GPT-3.5 answering over the top five Google results, in 2023.
For the API key page, the first paragraph should answer with a concrete fact. Before:
API keys are an important part of keeping your account secure. In this guide we’ll walk through everything you need to know about rotating them.
After:
To rotate an API key without downtime, create a second key, deploy it next to the old one, wait until the usage log shows no requests on the old key, then revoke it. AWS IAM, for example, allows up to two access keys per user, so both can be active during the switch.
The second version stands alone when quoted and carries a checkable fact. Don’t overcorrect: Google warns that content for every query variation made “primarily to manipulate rankings violates Google’s scaled content abuse spam policy,” and says structured data “isn’t required for generative AI search.” Keep schema accurate for rich results, but don’t expect it to carry a page.
Step 4: Get mentioned beyond your own site
Retrieval gets a page into the candidate pool; the model still picks whom to trust. Ahrefs' study of 75,000 brands found branded web mentions had the strongest correlation with AI Overview visibility (0.664), against 0.218 for backlinks. Correlation isn’t causation, but it fits fan-out: the more pages on your topic mention you, the more sub-queries you appear in. Pew found Wikipedia, YouTube and Reddit were the most-cited AI summary sources, so for a developer tool, solve real problems in Reddit threads and put talks on YouTube.
Step 5: Measure citations, not just rankings
Start with a mention tracker, because clicks undercount AI visibility: assistants sent 0.41% of visits in August 2026 across Ahrefs’ 112,309-site tracker. Lumirank asks your prompts, such as “how do I rotate an API key without downtime”, every day on ChatGPT, Gemini, Perplexity, Copilot, Google AI Mode and AI Overviews, logging whether you’re named, your rank against rivals, sentiment and the pages each answer cited. Its free plan covers 10 prompts on two engines (September 2026). Our AI search analytics roundup compares the alternatives.
Then cross-check with first-party data. Search Console’s Search Generative AI performance report, live for every site since August 31, 2026, counts how often your URLs appeared in AI Overviews, AI Mode and Discover’s AI features, by page, country and device; clicks stay in the regular Performance report under the Web search type. Bing's AI Performance report, in preview since February 2026, counts citations across Copilot and Bing’s AI summaries and lists “grounding queries”, the phrases the AI used when it retrieved your page.
For ChatGPT, OpenAI says it “automatically includes the UTM parameter utm_source=chatgpt.com in referral URLs.” In GA4, add a custom channel with a “matches regex” condition on session source, placed above Referral since GA4 uses the first channel that matches:
.*(chatgpt\.com|chat\.openai\.com|perplexity\.ai|claude\.ai|gemini\.google\.com|copilot\.microsoft\.com).*Clicks from AI Overviews and AI Mode arrive as google / organic, so GA4 can’t separate them. For the crawl side, count bot hits in your access logs, and check surprising spikes against the IP lists OpenAI and Anthropic publish:
grep -oE 'OAI-SearchBot|ChatGPT-User|GPTBot|Claude-SearchBot|Claude-User|ClaudeBot|PerplexityBot|Perplexity-User|Googlebot|bingbot' access.log \
| sort | uniq -c | sort -rnTesting an unpublished page with Pinggy
Google’s Rich Results Test and PageSpeed Insights need a public URL to render a page the way Google does. To check a staging build, serve it locally behind a tunnel:
# serve the built site (Hugo writes it to public/)
python3 -m http.server 8000 --directory public
# in a second terminal
ssh -p 443 -R0:localhost:8000 free.pinggy.ioThe tunnel prints a .run.pinggy-free.link HTTPS address that lasts 60 minutes. One catch: free tunnels show a one-time warning page to browser-style User-Agents, which in my test included Googlebot, Google-InspectionTool, Chrome-Lighthouse and ChatGPT-User (plain curl went straight through). For Google’s tools, use a Pro tunnel, which skips the page: ssh -p 443 -R0:localhost:8000 <token>@pro.pinggy.io, with a token from the Pinggy dashboard.
What to do this week
Run ai_bots.py and the word-count check against your five most important URLs, and fix any accidental block or near-empty HTML. Rewrite the first paragraph under each key heading so it answers the question with one checkable fact. Add your top questions to Lumirank, confirm Settings > Search generative AI in Search Console says Include, verify the site in Bing Webmaster Tools, and add the GA4 channel. A month later, the cited pages and grounding queries show which sub-questions you win, and what to write next.
Conclusion
AI search changed who reads your page first, not the plumbing underneath it. A model still finds its sources through a search index, robots.txt rules and the HTML your server returns, so most of these fixes help classic rankings too. What’s new is the scoreboard: citations and grounding queries now show which questions you’re winning. For the wider strategy, our GEO guide and agent-ready website checklist go deeper on crawler policy and entity signals.