AI search answers can be useful, cited and still unsafe to treat as reliable. A response may accurately quote one source, then add an unsupported comparison, a stale product detail or an invented policy condition. For SEO and content teams, the unit that matters is not simply the answer or the citation. It is the individual claim.
That distinction changes measurement. Counting citations tells you whether an engine displayed sources. It does not tell you whether each material statement is supported by an appropriate, accessible and current source. AI search hallucination measurement should therefore test the relationship between claims, evidence and user consequences.
This article proposes a practical convention for doing that. It is not a validated industry standard, and its numerical weights are not scientific estimates of truth or harm. Its value comes from consistent application across prompt sets, engines and reporting periods. Used that way, it gives teams a more defensible way to identify where unsupported answers create brand, customer and content risks.
Start with claims, not answer-level labels
An answer-level label such as “hallucinated” is usually too blunt for operational work. A single response can contain a supported definition, an unsupported performance promise and an ambiguous recommendation. Treating all three alike hides both the failure and the fix.
Break each answer into atomic, testable propositions. An atomic claim expresses one fact, comparison, recommendation, causal assertion or condition that a reviewer can check independently. Split compound sentences whenever their parts could have different evidence outcomes.
- “Brand X offers annual billing” is one factual claim.
- “Brand X is cheaper and better for enterprises” contains at least two claims: a price comparison and a suitability recommendation.
- “You can cancel at any time without a fee” may contain a cancellation-right claim and a fee claim.
Keep the answer text, prompt, engine, date, displayed citations, citation positions and captured result state with every claim. This provenance is essential when an answer changes or a reviewer disputes a label. The operating discipline closely follows the principles in AI Search Measurement Provenance: a score without a reproducible capture is an observation, not dependable measurement.
Define support as an evidence test
A citation next to a sentence is a lead, not proof. Reviewers should open the cited material and ask four questions: does it directly support the claim; is it authoritative for this claim type; is it current enough; and is the source accessible to the user and evaluator?
Accessibility deserves its own check. A source may be dead, blocked, paywalled or rendered in a way that prevents verification. Those conditions do not automatically make the underlying claim false, but they weaken what an answer visibly establishes. Google notes that AI features may use content subject to normal Search requirements and crawlability considerations, while publisher controls and access conditions can affect what systems can retrieve or show. See Google’s documentation on AI features in Search. For a measurement workflow, also test whether the cited evidence remains usable, as outlined in this citation accessibility framework.
A proposed A0–A5 support scale
The following scale is a proposed labelling convention. It deliberately separates absence of displayed evidence from contradiction. The order reflects practical triage: a direct contradiction normally demands faster action than an uncited but plausible statement. It should not be read as a universal taxonomy or a probability model.
| Code | Evidence status | Reviewer interpretation |
|---|---|---|
| A0 | Directly supported | An appropriate source explicitly substantiates the claim, including meaningful conditions. |
| A1 | Supported with limitation | The source supports the core statement, but the answer omits a material qualifier, scope or date. |
| A2 | Indirectly supported | Evidence permits a cautious inference, but does not state the answer’s stronger wording. |
| A3 | Unverified or uncited | No displayed or independently located adequate evidence supports the claim in the review window. |
| A4 | Weakly supported | The available source is stale, low-authority, inaccessible, irrelevant or materially incomplete. |
| A5 | Contradicted | A better or primary source directly conflicts with the answer. |
A0 and A1 are not interchangeable. If an answer says a benefit applies to every plan but the official page limits it to one tier, the core topic may be right while the decision-relevant condition is wrong. That is often a summary-fidelity issue rather than pure fabrication; use a parallel review of lost caveats and meaning where relevant.
Separate source gaps from model errors
Unsupported output has different causes, and lumping them together leads teams to fix the wrong thing. A content team cannot correct a model-generation failure merely by publishing another FAQ. Conversely, a missing or inaccessible primary source can make a sound claim difficult for a retrieval system to ground.
Add a cause label after assigning the A-status. The label is a reasoned diagnosis, not a claim about an engine’s private architecture.
| Cause label | Typical signal | Most plausible response |
|---|---|---|
| Source gap | No accessible authoritative page states the fact clearly. | Publish or improve a dated primary source with clear claim-level language. |
| Retrieval or accessibility gap | Strong evidence exists but is not retrieved, cited or available to the reviewer. | Check crawlability, rendering, canonicalisation, permissions and passage structure. |
| Attribution gap | The answer is accurate but lacks a visible citation, or cites an unrelated page. | Measure citation precision before treating the answer as grounded. |
| Transformation error | A cited source is present, but the answer adds, reverses or overstates meaning. | Document the mismatch; improve source clarity but do not assume publishing alone will solve it. |
| Staleness conflict | The answer follows an older source while newer primary evidence differs. | Update first-party materials, dates and internal links; monitor recurrence. |
This classification is intentionally cautious. Public results rarely expose the full retrieval and generation process, so “retrieval gap” means evidence consistent with retrieval failure, not proof of it. Perplexity’s published bot guidance, for example, is useful when checking whether access controls might affect collection, but it does not explain every answer outcome: Perplexity bot documentation.
Score risk by exposure, severity and user impact
Raw unsupported-claim rate is valuable, but it does not prioritise work. An uncited adjective in a low-visibility exploratory answer is not equivalent to a contradicted eligibility instruction in a highly visible recommendation. I recommend reporting both a rate and a risk-weighted queue.
Use the following proposed convention for each non-A0 claim:
Claim Risk Score = Support Weight × Visibility × Severity × User Impact.
- Support Weight (1–5): A1 = 1, A2 = 2, A3 = 3, A4 = 4, A5 = 5.
- Visibility (1–5): estimated prompt demand, answer-surface prominence, recurrence and whether the brand is named.
- Severity (1–5): factual centrality and degree of contradiction. A wrong price, policy, safety or eligibility statement ranks higher than decorative language.
- User Impact (1–5): the likely consequence if a reasonable user acts on the claim, considering money, time, safety, reputation and reversibility.
The 1–5 range is chosen for speed and reviewer calibration, not because five is objectively five times worse than one. Multiplication is useful because it prevents a low-impact issue from outranking a high-exposure contradiction merely through volume. Some teams may prefer an additive model when they need gentler differences between categories. Pick one, document it and resist changing thresholds mid-quarter.
For example, an answer that falsely says a brand’s service is available in a regulated region might receive A5=5, visibility 4, severity 5 and user impact 5, for 500. An uncited but low-stakes feature description at A3 with scores of 1, 1 and 2 receives 6. The numbers do not prove legal or commercial harm. They create a transparent order for investigation.
Build a review dataset that can withstand disagreement
Sample prompts across branded, non-branded, comparison, recommendation, troubleshooting and post-purchase intents. Include query variants and follow-up turns because an answer can become less grounded as the conversation narrows. Capture the full response rather than only the sentence mentioning your brand.
For each claim, record its exact wording, support label, cause label, cited URL, evidence excerpt, publication or update date, reviewer rationale and confidence. Store a snapshot or permitted archive reference where possible. A source inventory alone is insufficient; citation share of voice measures presence, whereas this framework tests whether the generated statement earns the confidence implied by its presentation.
Use two reviewers on a calibration subset, especially for A1, A2 and A4. Disagreement often reveals unclear definitions of “material qualifier” or “authoritative source,” not reviewer incompetence. Track agreement by category and revise the written decision rules before scaling. The process described in AI Search Evaluator Agreement is a useful companion.
Turn findings into content and monitoring decisions
Prioritise the highest-risk claims, then inspect their cause patterns. Source gaps call for durable first-party documentation: explicit eligibility, pricing scope, product limitations, effective dates and evidence close to the claim. Retrieval gaps call for technical and information-architecture checks, including whether a key passage is visible in rendered HTML and understandable out of context.
Do not “optimise” by adding unsupported marketing copy solely because an engine repeats it. The better response is usually to establish a precise, maintainable source of record and make its conditions unambiguous. For claims that are inherently variable, publish update dates and boundaries rather than broad evergreen assurances.
Monitor the unsupported risk rate alongside answer visibility, citations and framing. Segment by engine and intent. A falling aggregate rate can conceal a worsening recommendation segment, while a stable rate can hide a newly high-visibility contradiction. Review after major site changes, policy changes, model updates and seasonal changes in facts.
FAQ and conclusion
What is the difference between an unsupported claim and a hallucination?
An unsupported claim lacks adequate evidence in the review. A hallucination usually implies that the statement is invented or false. Keep them separate: A3 and A4 require investigation, while A5 records direct contradiction.
Should every uncited answer claim be treated as an error?
No. An answer can be correct without a visible citation. But absent evidence reduces verifiability, especially for consequential or changing facts. Label it, assess impact and avoid asserting that the engine made a proven model error.
How often should teams measure?
Run a stable core prompt set on a regular cadence and add event-driven checks after material changes. Consistency in prompts, capture method and labels matters more than an arbitrary frequency.
What should be reported to leadership?
Report unsupported-claim rate, total risk-weighted score, the highest-risk claims, recurring cause labels and remediation status. Show representative evidence excerpts, not only a dashboard total.
Conclusion: Claim-level AI search hallucination measurement will not reveal an engine’s internal reasoning or establish universal truth. It can, however, make answer quality reviews more useful when teams apply fixed definitions, preserve evidence and distinguish source weakness from transformation error. Treat the score as a prioritisation convention, revisit its assumptions, and focus remediation on the claims users are most likely to see and act upon.
