How we measure citation share, method v1.0

Every figure we publish carries a stamp naming the method version that produced it. This is that method. When it changes, the version changes, and figures published under an old version keep their old stamp rather than being quietly restated.

What citation share means here

Citation share is the proportion of answers, across a defined query set, in which a given company is cited as a source.

It is deliberately narrow. It is not share of voice, not sentiment, and not a composite index with weights we chose. Composite scores are easier to sell and impossible to audit, and the first time a client asks why their number moved, you want an answer that is a fact rather than a weighting decision.

We record three separate things per answer, because they move independently:

• Named. The company appears in the text of the answer.
• Cited. A URL on the company’s domain appears in the answer’s sources.
• Absent. Neither.

A company can be named constantly and cited never, which usually means the model knows of it from training and is not currently retrieving it. That distinction changes what you do about it, so we refuse to collapse the two into one number.

Query construction

Between 20 and 50 queries per engagement, 30 for an index category.

Three intent types, tracked separately:

Category queries. “Best commercial real estate underwriting software.” The ones where a buyer is choosing and has not decided. These carry the most weight because they are the shortlist question.

Problem queries. “How do I check development feasibility for a site.” Buyer has a problem and no vendor in mind.

Brand queries. “What is [company].” Used to test how a model describes a company, and to catch hallucinations. These are the only type that scales per company.

Queries are written in the language buyers use, not the language vendors use. They are approved by the client before anything runs, because the query set is the contract. Changing the query set mid-engagement invalidates the comparison, so we do not.

Engines and runs

Five engines: ChatGPT, Claude, Gemini, Perplexity, and Google AI Overviews.

Three runs per query per engine. Every query is run logged out, without personalisation, from a consistent location.

Three runs is a floor, not a comfort. Generated answers vary between runs, and a single run is an anecdote. Three tells you what a model tends to say. It does not give you a confidence interval, and we do not report one, because reporting statistical confidence from three samples would be dressing up a small number.

Where the three runs disagree substantially, that disagreement is itself a finding and gets reported rather than averaged away.

What we store

One row per run, per query, per engine, per company, carrying: run id, query, engine, run index, company, mention type, cited URL, position, and timestamp.

We store the raw response, not only the parsed result.

This is the design decision that matters most and it is the one most commonly skipped. Parsers are wrong sometimes. Brand names collide with ordinary words, a competitor gets missed, a citation is attributed to the wrong clause. If only the parsed output was kept, a parser fix cannot be applied backwards and the history is permanently built on the old mistake. Because the raw answer is stored, a correction re-reads history instead of re-running and re-paying for it.

We also store a hash of each response. When an engine’s answer to a query has not changed between runs, that is worth knowing, and it is occasionally worth publishing.

Parsing

Citations are extracted from the structured source lists the engines return where they return them, which is most of them. Company mentions are matched against a name list including known aliases, legal entity names, and product names, because “Archer” and “Archer Technologies” and the product name are the same company and a naive match gets this wrong in both directions.

Mentions inside a cited page title, rather than in the model’s own prose, are recorded separately and not counted as named.

Where this method is weak

Four places. All of them are known, none of them are solved, and we would rather publish them than have a client find them.

The API and consumer app gap. We query most engines through APIs. Buyers use the consumer applications, which have different system prompts, different retrieval behaviour, personalisation, and sometimes a different model version. We spot-check roughly ten percent of results through a real browser session to gauge the size of the gap, and we report when the gap is large. We cannot eliminate it.

Google AI Overviews fidelity. AI Overviews requires a search results provider rather than a model API. Vendor claims about fidelity conflict with each other. We trial providers against a set of known queries and inspect the parsing manually before trusting a provider, and we treat this engine’s numbers as the least reliable of the five.

Three runs is a small sample. Enough to distinguish “usually” from “once”. Not enough to detect a small change. We do not report movements we cannot distinguish from run-to-run variance, which means some real small movements go unreported.

Model updates confound everything. An engine can change retrieval behaviour overnight with no announcement. A change in a client’s numbers may be our work, may be a model update, and frequently we cannot separate them. Where we cannot, we say so rather than claiming the credit.

What this measurement cannot tell you

It cannot tell you what a citation is worth.

Attribution from generated answers is unreliable. Many answers produce no click. The buyer who reads your name in an answer and searches for you directly a week later arrives as brand search. We do not model this and we do not quote pipeline figures derived from it.

Citation share is a leading indicator of whether you are in the consideration set when a buyer asks a model. That is a real thing to care about, and it is a different thing from revenue. Treating it as the second is where this whole category loses its credibility.

Changes from previous versions

None. This is v1.0.