GEO: Does Your Site Show Up in AI Answers? A 21-Bot Access Matrix, llms.txt and Log Analysis

In classic SEO the question was clear: can Google crawl, index and rank your page? Now there's a second question next to it: when someone asks ChatGPT, Claude or Perplexity about your industry, do you show up as a source in the answer? That's what GEO (Generative Engine Optimization) is about. In Yankı, the SEO + GEO platform I'm building, I set out to make the technical side of that question measurable first. Content quality matters, but if the bot can't get into your site at all, it doesn't matter how good the content is.

There's no single "AI bot": three distinct classes

The first thing I noticed is that "should we block AI bots?" is the wrong question. If you read the providers' official documentation, bots serve three different purposes, and I split the catalog into those three classes:

• Search/answer bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot...): the indexing bots of AI search engines. Block them and your site won't appear as a source in answers.
• User-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User...): bots that fetch a page on the spot when someone asks a question. They don't crawl automatically, and some providers say outright that these bots may not always follow robots.txt.
• Training bots and control tokens (GPTBot, ClaudeBot, CCBot, Google-Extended...): they collect content for model training. Blocking them doesn't affect your search visibility.

That distinction changes a lot in practice. "I don't want AI trained on my content, but I do want to show up in ChatGPT search" is a perfectly consistent preference: you close the training bots and keep the search bots open. The problem is that many sites do this wholesale with a single Disallow block and unknowingly shut off their search visibility too.

The catalog has 21 bots and is versioned: when the list changes, CATALOG_VERSION goes up, and a result computed with an older version says so on screen. Every source is the provider's own bot documentation, because lists compiled from blog posts go stale very quickly in this space.

The robots.txt access matrix: not as simple as it looks

For each bot I evaluate robots.txt with real parsing rules and produce a matrix: can this bot reach the home page and the important paths? There are two subtle points here.

The first is fallback behavior. Applebot, for example, if robots.txt has no group of its own, follows the Googlebot group before falling back to *. If you don't model that, a site you report as "open to Apple" can actually be closed. The catalog stores this relationship as a field for each bot. The robots.txt library I use had a trap here too: it also matches a group name as a prefix of the bot name, so it treats an Applebot group as applying to Applebot-Extended. Under RFC 9309, group selection must use the full product name. So I pick the group myself: the exact-name group first, then the catalog's fallback group, then *.

The second is control tokens. Google-Extended and Applebot-Extended aren't separate crawlers; they're permission names used only in robots.txt. They never visit the site and never appear in a User-Agent. They're evaluated in the matrix but not searched for in log analysis. Otherwise we'd produce a meaningless warning like "Google-Extended has never visited your site".

llms.txt: does it exist, and is it really llms.txt?

/llms.txt is a proposed format for a summarized map of a site for AI models: a single H1 title, a short summary prefixed with >, then link lists under H2 headings. The audit checks whether the file exists, its structure (no H1 / H1 not first / multiple H1s, no summary, no sections, no links) and its size. Since the summary file is meant to fit in a context window, large files get a warning.

The check that turned out most useful came from somewhere I didn't expect: soft 404s. Many sites answer a nonexistent path not with a 404 but with a 200 and the home page's HTML. A naive check would say "llms.txt exists". Yankı looks at the content type and the first bytes and flags it as not_text. The file's content is also treated as external data: it's only parsed, and the short excerpts shown on screen are rendered with template escaping.

Are the bots actually coming? Server log analysis

robots.txt shows intent; logs show reality. The user uploads an Apache/nginx access log, and Yankı extracts how many requests each bot made per day, which status codes it got, and which paths it requested most. A finding like "PerplexityBot shows up, but half its requests get a 403" usually means a WAF or CDN rule is quietly cutting off AI bots, which is why the CDN/WAF detection sits on the same screen.

Log files are large and untrusted input, so the analysis is streaming: the file isn't kept in memory or on disk; it's processed chunk by chunk. The format is determined by the first bytes, not the extension. gzip (including multi-member files) is decompressed, but the decompressed size is capped, which protects against compression bombs. Other known archive formats are rejected. Malformed or overly long lines are counted and skipped without stopping the run.

On the privacy side I set a hard rule: IP addresses, usernames, referrers and non-bot requests are never stored. Only bot × day × status-class counts and each bot's most-requested paths (with query strings stripped) are kept. There's also an honest limitation: User-Agents can be spoofed, and there's currently no reverse-DNS or IP-range verification. The screen says so explicitly. A tool admitting what it doesn't know is worth more than one that overstates what it does.

The crawler has to play by the rules too

A tool that audits other people's sites shouldn't itself be a badly behaved bot. Yankı's crawler respects robots.txt and Crawl-delay. If robots.txt doesn't allow Yankı, it won't even request llms.txt; the result shows as blocked. At the network layer, a dedicated client prevents a user-supplied address from being pointed at the server's internal network or a cloud metadata service (SSRF): IP pinning, manually followed redirects, rejection of private address ranges and capped gzip decompression.

Wrapping up

The content side of GEO is still very much debated, but the technical side is measurable: are you open to the right bots, is your llms.txt really llms.txt, and what response do bots get when they arrive? In Yankı I tried to answer those with a classified, versioned bot catalog, a robots matrix that follows the real rules, and privacy-preserving log analysis. The 51-rule technical SEO audit and the multi-tenant architecture are on the project page.