An AI search evaluation dataset is the control surface for serious answer-quality measurement. Without one, teams tend to compare screenshots, react to memorable failures and mistake a model’s fluent answer for a correct one.
The dataset should not be a collection of preferred model outputs. It should be a versioned set of queries, answer requirements, source-backed claims and explicit scoring rules. That distinction matters. A model response is the object being tested; it is not the evidence that establishes truth.
For SEO and content teams, this creates a usable bridge between editorial governance and AI search visibility. You can test whether an answer is accurate, complete, appropriately cautious and supported by citations. You can also identify whether a weak result reflects your content, source selection, a changing model or an evaluation rule that needs refinement.
Start with a decision, not a list of keywords
Build the dataset around decisions the audience needs to make. A broad keyword export is useful input, but it is not a defensible test set. Queries should represent the work users expect an answer engine to do: explain, compare, recommend, calculate, troubleshoot or provide current guidance.
Define the population first. For example: non-branded UK queries about enterprise payroll software, in English, where a buyer needs factual implementation guidance. Then define the unit of evaluation: one fixed prompt, one locale, one date or freshness window, and one expected evidence standard.
Keep prompt wording stable during a measurement period. Record the exact prompt, model or product, access date, signed-in state, location, language, conversation history and tool settings where observable. Those variables can change results materially. For a broader measurement structure, see this AI search measurement taxonomy.
Sample queries to represent intent and risk
A compact, stratified sample is usually more useful than a large, unstructured one. Divide candidate queries by intent, funnel stage, topic, geography, freshness sensitivity and consequence of error. Include the queries that generate commercial attention, but do not exclude awkward questions merely because your site does not yet answer them well.
High-risk categories deserve deliberate coverage: health, finance, legal, safety, pricing, compatibility and time-sensitive policy questions. They reveal whether an engine invents certainty, cites weak sources or should decline to answer.
| Field | Why it belongs in the dataset |
|---|---|
| Query ID and exact prompt | Makes reruns and audit trails possible. |
| Intent and task type | Prevents comparison of unlike answer expectations. |
| Audience, market and language | Captures legitimate variation by context. |
| Freshness class | Sets the review date and expiry rule for evidence. |
| Risk tier | Determines source standards and escalation needs. |
| Expected claim set | Provides the denominator for completeness. |
Use random sampling within each stratum where possible. Then add a small, named “sentinel” set: critical product claims, recurring incidents and known ambiguity traps. Do not quietly replace failed prompts with easier ones. Retain them and document why they matter.
Create reference answers from sources, not generated prose
A reference answer is not necessarily a polished paragraph. In an evidence-aware dataset, it is a structured claim inventory tied to source passages. Start with primary sources: official documentation, regulator publications, standards bodies, original research methods and manufacturer specifications. Use reputable secondary sources when primary evidence does not answer the question, and label that limitation.
For each source, capture publisher, title, URL, publication or update date, access date, relevant excerpt, source tier and applicability conditions. A claim about a feature available only on one plan or in one country is not a universal claim.
Google’s documentation on AI features and your website is a useful primary reference for questions about Google Search participation. But it should not be stretched into evidence about other answer engines. Likewise, source accessibility can affect what systems retrieve; Perplexity publishes its crawler guidance at its bots documentation.
Where sources disagree, preserve the disagreement. A valid reference may say that evidence is mixed, jurisdiction-dependent or insufficient. Forcing one synthetic “correct” answer can make evaluators punish appropriate uncertainty. The practical handling of competing evidence is covered in this framework for sources that disagree.
Write atomic, testable claims
Break the expected answer into atomic claims: each should express one proposition that a reviewer can verify against a passage. “Plan X includes feature Y” and “Feature Y is available in the UK” are separate claims. So are a recommendation, an exception and a date qualification.
For every claim, assign one of these reference labels:
- Required: needed for a materially complete answer.
- Optional: useful detail, but not necessary to fulfil the task.
- Conditional: required only when stated conditions apply.
- Unresolved: evidence is inadequate or conflict cannot responsibly be reconciled.
- Abstention expected: the correct response is to state a limit, request missing context or avoid a determination.
Assign weights sparingly. A safety-critical exception can reasonably count more than a minor explanatory detail, but complex weights often conceal inconsistent editorial judgment. In my experience, clear required/optional rules and a short rationale are more durable than elaborate scoring formulas.
Label answer claims separately from citations
One answer can be factually correct without a citation, cited but unsupported, or complete but overconfident. Do not collapse these into a single “good answer” label.
At claim level, reviewers should label whether the model claim is supported, contradicted, not verifiable from the reference set, too vague to assess or unsupported addition. “Not verifiable” is not automatically false; it means the approved evidence set does not establish it and it needs source review.
For cited claims, assess citation entailment: does the cited source actually support the specific proposition, including quantities, conditions and time frame? A nearby citation or a credible domain is not enough. Citation correctness has its own mechanics, explained in this evidence-entailment framework.
| Label | Operational rule |
|---|---|
| Entailed | The cited passage supports the claim as written. |
| Partial entailment | The source supports only part of the claim or misses a material condition. |
| Non-entailing | The source does not support the claim, even if it is topically related. |
| Unavailable | No usable citation or source passage is supplied. |
Score completeness against eligible required claims, not every claim ever listed. If a query does not provide country or product-plan context, a conditional claim may be ineligible. This denominator discipline avoids treating relevant clarification as a failure.
Define abstention before you score it
Abstention is often mishandled because teams count every non-answer as failure. A useful abstention can protect users when evidence is unresolved, the question lacks material context or the requested conclusion exceeds available evidence.
Label the expected behavior in advance: answer directly; answer with qualifications; ask a clarifying question; or abstain. Then label the observed behavior as appropriate, overbroad, insufficient or unsafe. An engine that supplies a confident recommendation where the dataset requires a clarification has not earned completeness credit.
This is particularly important in visibility reporting. Absence, refusal and omission are distinct outcomes, not interchangeable zeros. Read how to measure AI search visibility when models abstain, refuse or omit answers for the measurement implications.
Worked example: calculate the result from fixed labels
Suppose the fixed query is: “Does Product A support SSO for UK customers?” The reference record, verified on a stated date, contains three claims: C1, Product A supports SSO (required, weight 2); C2, SSO is available to UK customers (required, weight 1); and C3, SSO requires the Enterprise plan (conditional, weight 1, eligible only if plan is discussed). The primary documentation is the reference source.
| Response | Claim labels | Citation label | Eligibility and result |
|---|---|---|---|
| A: “Yes. UK customers on Enterprise can use SSO.” | C1 supported; C2 supported; C3 supported | Entailed | All three claims eligible. Completeness = 4/4 = 100%. |
| B: “Yes, Product A supports SSO.” | C1 supported; C2 omitted; C3 omitted | Unavailable | Plan is not discussed, so C3 is ineligible. Completeness = 2/3 = 66.7%. |
| C: “All Product A customers can use SSO at no extra cost.” | C1 supported; unsupported addition contradicts C3 | Non-entailing | C2 omitted; C3 eligible because plan/access is asserted. Completeness = 2/4 = 50%; contradiction rate = 1/3 observed claims. |
For a three-response run, mean completeness is (100 + 66.7 + 50) / 3 = 72.2%. Citation entailment is 1 of 2 supplied, assessable citations = 50%. Report the unsupported contradiction separately; do not let a respectable completeness average hide it. The example also shows why denominator eligibility must be written before reviewers see model output.
Make human review reproducible
Use two independent reviewers for a meaningful subset, especially high-risk queries and newly introduced labels. Give them the same frozen source pack and a codebook with positive and negative examples. Blind reviewers to model identity when feasible; it reduces brand expectations from leaking into labels.
Measure agreement by label type. Simple percentage agreement can be useful for operations, while Cohen’s kappa or a comparable chance-adjusted statistic can add context for categorical labels. Neither number substitutes for adjudication notes. Low agreement may reveal an ambiguous rubric, unstable source evidence or a claim that is too broad—not merely reviewer error.
Keep an adjudication log: disputed item, each initial label, final decision, rationale, owner and codebook change. Version the dataset whenever sources, rules or expected claims change. Freshness-sensitive topics need scheduled revalidation, not an annual cleanup. This aligns naturally with an evidence inventory and named claim owners; see AI search content governance for claims, evidence and freshness.
FAQ and conclusion
FAQ
How many queries are enough? Start with enough queries to cover every meaningful intent, risk tier and market segment. Expand after agreement is stable; an unreviewable large set is weaker than a smaller, governed one.
Can our own content be a reference source? Yes, when it is the authoritative source for a first-party product fact. It should still be versioned, attributable and checked for conditions. It cannot independently validate broad industry claims.
Should a model receive points for useful uncited advice? Score usefulness separately if it matters to your use case. Do not treat uncited advice as verified factual support.
Conclusion
A defensible AI search evaluation dataset makes answer testing inspectable rather than impressionistic. Fix the prompt and context, ground expected claims in dated evidence, score claims and citations independently, use eligible denominators, and preserve appropriate abstention. The resulting metrics will not make answer engines predictable, but they will make your decisions more accountable.
