AI search reporting goes wrong when five very different things are called “visibility.” A page may be eligible for retrieval but never used. A brand may appear in an answer without a link. A cited page may support one minor claim but not a recommendation. And a recommendation may influence a later purchase without producing a trackable click.
Those are not interchangeable outcomes. Treating them as such creates attractive dashboards and unreliable decisions.
An AI search measurement taxonomy gives each outcome a separate definition, evidence standard and reporting destination. It does not promise access to every internal system decision. In most answer engines, teams observe outputs and infer only what the evidence permits. That constraint is healthy: it forces a distinction between what was seen, what was likely and what was actually attributable.
Start with five distinct outcome states
The useful unit of analysis is not “did we rank?” It is a query run: a defined prompt, engine, locale, device context, time, response and source set. Within that run, record five states.
| State | Operational definition | Strongest practical evidence | What it does not prove |
|---|---|---|---|
| Retrieved | A first-party URL or domain entered the engine’s candidate or retrieved source set. | Engine-provided retrieval data, inspection tooling or a clearly exposed source panel. | That the source affected the final answer. |
| Included | The brand, product, entity or claim appears materially in the answer text. | Saved answer capture with a human or rules-based annotation. | That the answer relied on your content. |
| Cited | A first-party URL or domain is attached as a source to an answer claim or answer section. | Captured citation target, placement and supported claim. | That users noticed, trusted or clicked it. |
| Recommended | The engine presents the brand or product as a positive option for the user’s stated need. | Answer wording, position, qualifiers and comparison context. | That it was the best option, or that a user will choose it. |
| Converted | A measurable downstream outcome occurred and meets a documented attribution rule. | Analytics, CRM or experiment data joined to a defined exposure signal. | That AI search alone caused the outcome. |
Retrieval is often the least observable state. Some interfaces expose citations without exposing all sources considered; others expose neither. Do not label a URL “retrieved” simply because it was cited. A citation is evidence of use in the visible answer, not a complete retrieval log. When retrieval cannot be verified, report retrieval status: unknown.
Inclusion is also broader than a bare string match. “Acme” in a navigation list or a passing comparison is not equivalent to a substantive answer to the user’s question. Use an inclusion threshold: the brand must be named or unambiguously resolved, and it must be connected to a relevant attribute, task or claim.
For a deeper treatment of prominence, including recommendation strength and answer position, see AI Search Brand Prominence Measurement.
Use an evidence ladder, not a single visibility score
A single composite score is tempting because it simplifies executive reporting. It also hides the problem. A 60% “AI visibility” score could mean frequent brand mentions with no citations, a small number of high-quality citations, or many citations that never lead to qualified traffic. Those require different work.
Keep the states separate in the source data, then use a funnel view for discussion:
- Source opportunity: verified retrieval rate, where observable, and citation eligibility diagnostics.
- Answer presence: inclusion rate and qualified inclusion rate.
- Evidence presence: citation rate, cited-claim coverage and citation accuracy.
- Decision influence: recommendation rate, recommendation position and comparison-set presence.
- Commercial outcomes: assisted conversions, direct referrals and qualified pipeline where applicable.
Each rate needs its own denominator. Citation rate is cited query runs divided by eligible, successfully captured query runs. Recommendation rate is recommendation-positive runs divided by runs with a recommendation-seeking intent. Conversion rate is not divided by prompts at all; it is reported against a defined exposed or referred audience, with attribution limitations stated beside it.
Intent segmentation matters. A user asking for a definition does not create the same recommendation opportunity as one asking which provider to choose. Build prompt cohorts before drawing comparisons. This is closely related to measuring AI search visibility by query intent.
Capture events that another analyst can audit
The schema should preserve raw evidence before it creates a metric. Store a response capture or permitted rendered record, not merely a vendor score. Hash the capture, retain the timestamp and keep the exact prompt text. This makes later annotation review possible when model behaviour changes.
| Event | Required fields | Useful optional fields |
|---|---|---|
| query_run | run_id, prompt_id, engine, model or product surface, locale, timestamp, capture_status | device, logged-in state, experiment cohort, response hash |
| retrieval_observation | run_id, domain, URL, observation_method, status | rank or source order, retrieval confidence |
| answer_inclusion | run_id, entity, answer span, inclusion_type, annotator or rule version | prominence, sentiment, answer position |
| citation_observation | run_id, cited URL, cited domain, cited claim span, citation position | link destination, citation type, entailment judgment |
| recommendation_observation | run_id, entity, recommendation class, supporting wording | ranked position, qualifiers, competing entities |
| conversion_touch | user or session key, outcome, timestamp, attribution model, consent status | referrer, campaign parameters, CRM stage, exposure confidence |
Use controlled vocabularies. For example, recommendation class can be explicit first choice, shortlisted option, conditional fit or mere mention. Do not allow analysts to decide case by case after seeing commercial results.
Citations need claim-level review. A source URL can be present while failing to substantiate the sentence beside it. Record whether the cited page entails the claim, only partially supports it, contradicts it, or cannot be assessed. The method in How to Measure AI Search Citation Accuracy is a practical companion to this taxonomy.
Build reporting layers for different decisions
One dashboard should not serve everyone. Raw operational evidence, editorial actions and commercial interpretation belong in different layers.
Layer 1: observability and data quality
Report capture success, duplicate-run rate, unavailable citations, annotation agreement and the share of runs where retrieval is unknown. If capture quality falls, apparent visibility changes may be measurement noise. Also show prompt coverage by intent, market and product line.
Layer 2: answer and source performance
Show inclusion, citation and recommendation rates by query cohort. Add source-level measures: domains cited, URLs cited, claims supported and source concentration. A domain cited repeatedly for one narrow fact is not equivalent to broad topical authority.
Where source systems vary by day or locale, trend results with sample counts rather than treating every weekly movement as a meaningful change. Repeated sampling is especially important for non-deterministic outputs.
Layer 3: content and evidence actions
Turn gaps into owners and actions. Low inclusion with strong first-party citations may indicate weak entity association or poor answer framing. Frequent inclusion but low citation may indicate that the engine knows the brand from other sources while first-party evidence is not selected. Citations that support outdated claims call for correction, consolidation or stronger evidence blocks.
Structured data can clarify page entities and properties, but it is not a citation guarantee. Google’s structured data guidance is useful for implementation discipline; it should not be turned into a causal claim about AI answer inclusion. Likewise, review platform-specific documentation, such as Google’s AI features in Search guidance, for current crawling and appearance information.
Layer 4: business contribution
Separate direct referrals from assisted influence. Direct measurement may include tagged visits from a cited link. Assisted measurement might join opted-in survey responses, CRM self-reported discovery fields, controlled geographic tests or exposure cohorts to later outcomes. Each method has bias. Report the method, observation window and counterfactual limitations alongside the result.
For cases where citations do not generate a referrer, see How to Measure AI Search Assisted Conversions When Citations Send No Clicks.
Prevent inflated visibility claims
My professional rule is simple: describe the highest state directly evidenced, not the most commercially desirable state in the chain. “Our domain was cited in 18 of 100 captured runs” is auditable. “AI search drove awareness” is a hypothesis unless a separate study supports it.
- Never add mentions and citations into one numerator. A citation may include a mention, so this double-counts.
- Do not call third-party citations “brand citations” unless the cited source is first-party. Report earned-source presence separately.
- Do not treat a list placement as a recommendation without language or context that signals suitability.
- Do not infer retrieval from a missing citation, or infer non-retrieval from a missing mention.
- Label human-coded judgments, automated classifications and engine-reported data differently.
Freshness needs the same discipline. For each claim sampled, assign 0 when current and adequately evidenced, 1 for a warning (for example, an aging source or a claim needing verification) and 2 for a confirmed failure, such as a materially outdated or incorrect time-sensitive claim. The reproducible Freshness Failure Rate is sum of claim points divided by (2 × number of assessed claims) × 100. A warning therefore contributes one point; a confirmed failure contributes two. Exclude unassessable claims from the denominator and report their count. This preserves the 0–100 scale without pretending warnings and failures are equal.
FAQ and conclusion
Is a citation more valuable than a mention?
Not automatically. A citation is stronger evidence that a first-party source was used in the visible answer. A prominent, accurate recommendation may carry more decision influence than a minor citation. Measure both, then interpret them against query intent and answer context.
Can we measure retrieval when the engine does not expose it?
Usually not directly. Mark retrieval as unknown, measure observable citations and inclusions, and use controlled tests only to investigate hypotheses. Do not convert a correlation between a page change and a citation change into proof of retrieval.
What is the minimum viable reporting set?
Start with a representative prompt set, captured answers, inclusion rate, first-party citation rate, recommendation rate for relevant intents, and a data-quality panel. Add conversion reporting only after agreeing on an attribution method and consent-aware data handling.
Conclusion: AI search measurement becomes useful when it stops using visibility as a catch-all. Retrieval concerns candidate access. Inclusion concerns answer presence. Citation concerns sourced evidence. Recommendation concerns expressed suitability. Conversion concerns a separately observed business outcome. Preserve those boundaries in the event model, use denominators that match each question, and state uncertainty plainly. The result is less theatrical reporting, but much better editorial, technical and commercial decisions.
