AI search visibility experiments are easy to run badly. Change a page, submit a few prompts, see a new citation and declare success. The sequence feels persuasive, but it does not show that the change caused the result.
AI answers vary across prompts, sessions, locations, retrieval events and model versions. A source may appear in one response and disappear in the next without any change to the underlying page. Personalisation and product updates add further uncertainty. Even when a citation appears after an edit, the timing may be coincidental.
The right response is not to abandon testing. It is to use a protocol proportionate to the evidence available. Most content teams cannot run laboratory-grade experiments on commercial answer engines, but they can make comparisons more controlled, preserve a useful audit trail and avoid claims their data cannot support.
This protocol is designed for SEO practitioners, editorial teams and analysts testing changes such as answer-first summaries, clearer entity descriptions, original evidence, improved sourcing, internal links or crawl access. It focuses on two observable outcomes: whether a page is included in an AI answer and whether it is cited as a source.
Start with a narrow causal question
Do not begin with “How can we improve AI visibility?” That is a programme objective, not an experimental question. A workable test isolates one intervention, one content population and a defined set of outcomes.
A stronger question is: “For eligible product-comparison pages, does adding a concise, sourced decision summary increase citation frequency across a fixed sample of commercial-research prompts over six weeks, relative to comparable unchanged pages?”
This wording forces several decisions:
- Which pages are eligible?
- What exactly will change?
- Which prompts represent the intended demand?
- What counts as inclusion or citation?
- How long will observation continue?
- What comparison will be used?
Write the hypothesis and analysis plan before editing pages. Pre-specification does not eliminate bias, but it makes post-hoc storytelling easier to detect.
Choose one primary outcome. Citation frequency is often more auditable than a subjective judgment of brand prominence. Secondary outcomes might include answer inclusion without a link, citation position, supported claim coverage or referral visits. If downstream traffic matters, treat that as a separate measurement problem; see how to test whether an AI citation sends useful traffic.
Define the experimental unit and treatment
The experimental unit is the item assigned to treatment or comparison. For content tests, that is usually a page, page cluster or topic—not an individual AI response. Responses are repeated observations of an underlying unit.
Page-level assignment works when pages are sufficiently independent. Cluster-level assignment is safer when treatment can spill across closely linked pages. For example, changing a hub page may affect retrieval or interpretation of every supporting article. Treating individual pages in that cluster as independent would understate the relationship between observations.
Document the treatment as an editorial specification, not a vague instruction to “optimise for AI.” Include the sections added or removed, expected word range, source requirements, markup changes, internal links and elements that must remain untouched. Keep a versioned copy of every treated and comparison page.
Where possible, make one substantive change class per experiment. If you simultaneously rewrite the introduction, add statistics, change schema, improve crawl access and acquire links, the bundle may be testable, but the contribution of each component will not be identifiable.
Check whether the page can be retrieved
Before testing presentation changes, check indexing, canonicalisation, robots directives, rendering and relevant crawler access. A blocked or inaccessible page creates a different problem from an eligible page that is not selected as a source.
Google’s documentation on AI features and websites is useful for understanding its published search requirements, while Perplexity documents its declared bots in its bot guidance. These documents help with operational checks. They do not provide a formula for earning inclusion.
Build the prompt sample before the test
A convenient handful of prompts is not a defensible sample. Prompt wording changes retrieval and answer composition, so the prompt set is part of the experimental design.
Build a sampling frame from observable demand sources: search-query data, on-site search, customer-support themes, sales questions, research interviews and structured query expansion. Group near-duplicates into prompt families representing the same underlying task. A family might contain “best payroll software for restaurants” and “restaurant payroll platforms,” but not a separate task such as implementation guidance.
Stratify the frame using dimensions relevant to the test, such as intent, funnel stage, audience, geography, product category or question type. Set quotas before sampling so a large informational stratum does not overwhelm smaller but important commercial or troubleshooting strata.
Use a documented within-stratum selection rule
Apply the same selection procedure inside every stratum:
- Deduplicate near-equivalent prompts into families, preserving the variants and the reason they were grouped.
- Set pre-approved quotas for each stratum before looking at test outcomes.
- List all eligible prompt families and assign each a stable identifier.
- Randomly select eligible families until the quota is filled, using a recorded random seed or another reproducible method.
- Select one canonical wording and any planned variants from each chosen family according to a fixed rule.
- Log exclusions, replacements and missing strata with specific reasons.
If no defensible sampling frame or selection process exists, the set cannot be called representative. Call it a monitored convenience sample and narrow the conclusions accordingly. That wording matters. It separates useful directional monitoring from claims about a broader market.
For a fuller workflow, use this guide to building a representative prompt set for AI search visibility tracking.
Choose a control structure that matches operational reality
A randomised holdout is generally the clearest practical design. Identify eligible pages, pair or stratify them by topic, baseline visibility, authority, format and age, then randomly assign units to treatment or control. Leave controls materially unchanged during the observation window.
Random assignment reduces selection bias, but it does not guarantee balance in a small sample. Report baseline differences rather than assuming randomisation solved them.
When a permanent holdout is unacceptable, consider a staggered rollout. Randomly assign when eligible groups receive the change, so later groups temporarily serve as comparisons for earlier groups. This design is operationally attractive because every group eventually receives the treatment, although time trends and carryover effects require care.
A matched comparison is weaker but sometimes unavoidable. Match treated pages to unchanged pages using criteria selected in advance. Do not choose controls after seeing which ones produce the preferred result.
| Design | Best use | Main limitation |
|---|---|---|
| Randomised holdout | Enough comparable pages and permission to leave some unchanged | Small samples may still be imbalanced |
| Staggered rollout | All eligible pages must eventually receive treatment | Time effects can overlap with treatment |
| Matched comparison | Random assignment is not feasible | Unmeasured differences remain plausible |
| Interrupted time series | One major page or site-wide intervention | Model and market changes can mimic an effect |
A simple before-and-after test on one page is not controlled. It can still produce a useful case observation, but causal language should remain tentative.
Standardise collection without hiding real variation
The goal is not to force AI systems to return identical answers. It is to measure variation consistently.
For every observation, record the engine or product, displayed model if available, date and time, account state, location, device or interface, prompt text, conversation state and whether browsing or retrieval was enabled. Start fresh sessions unless conversational context is an intentional test factor.
Run repeated observations at pre-set intervals. Repetition is essential because a single response can be an unstable draw. Spread runs across time rather than issuing all repetitions in one short batch, which may overrepresent a temporary retrieval state.
Automation can improve consistency, but only where it complies with the product’s terms and technical constraints. Manual collection may be slower, yet it makes interface changes and citation presentation easier to inspect. Whichever method you choose, retain raw answer text, source URLs, screenshots where permitted and parsing logs.
Use explicit coding rules
Create a codebook before reviewing post-treatment results. At minimum, distinguish:
- Direct citation: the tested URL is shown as a source.
- Domain citation: another URL on the same domain is cited.
- Unlinked inclusion: the brand or content claim appears without an attributable link.
- No inclusion: neither the page nor defined entity appears.
- Unusable observation: error, refusal, failed retrieval or ambiguous output.
Define URL normalisation rules for parameters, fragments, redirects and canonical equivalents. Decide whether citation carousels, footnotes and inline links count identically. If answer quality is scored, use anchored criteria and have a second reviewer code a sample without seeing treatment status.
Do not silently discard failed or inconvenient runs. Mark them and report the rule used to include or exclude them.
Separate prompt variation from treatment effects
Use the same core prompt sample for treatment and comparison units. Where a prompt can retrieve multiple eligible pages, analyse at the prompt-family-by-time level and note competing URLs. If every page has a unique prompt set, page differences become tangled with prompt differences.
Prompt variants can be useful, but they should be treated as a planned factor. For example, each family might include a short query, a detailed natural-language question and a constrained follow-up. Rotate them using a fixed schedule rather than inventing wording during collection.
Keep an untouched validation set if the content inventory is large enough. Use the main sample to assess the treatment, then check whether the direction appears in prompts not used to shape the edit. Validation does not rescue a weak design, but it reduces the risk of optimising narrowly for known wording.
Track external events and contamination
AI search experiments occur in moving systems. Maintain an event log covering known product releases, interface changes, indexing incidents, site migrations, major competitor changes and relevant news events. A visible step change across treatment and control groups may indicate a shared external cause.
Also track internal contamination:
- Unplanned edits to control pages
- Site-wide template or navigation changes
- New internal or external links
- Canonical and redirect changes
- Publication of overlapping content
- PR activity or brand-demand spikes
Internal links deserve special attention because a treatment page can alter the discoverability or context of nearby controls. If internal linking is the intervention, specify the source pages, anchor policy and destination set in advance. The evidence-aware framework for AEO topic-cluster internal linking provides a useful planning model.
Analyse changes, not isolated post-test rates
The basic comparison is the change in the treatment group relative to the change in the control group. This is more informative than comparing post-treatment citation rates alone because the groups may have started at different levels.
For each group, calculate the proportion of valid observations with a direct citation during baseline and post-treatment periods. Then compare the two changes. The same structure can be applied to unlinked inclusion or domain-level citations, but keep the pre-declared primary outcome prominent.
Aggregate carefully. Hundreds of repeated runs do not necessarily equal hundreds of independent observations. Responses for the same prompt, page and period are clustered. Reporting every run as an independent unit can make uncertainty appear smaller than it is.
At a minimum, present results by page, prompt family, engine and observation period. For larger programmes, use an analyst familiar with repeated-measures or hierarchical models. For smaller tests, show the underlying counts and distributions rather than relying on a single percentage.
| Result layer | What to report |
|---|---|
| Coverage | Eligible pages, sampled families, engines and valid runs |
| Baseline | Group-level citation and inclusion rates before treatment |
| Change | Treatment change compared with control change |
| Consistency | Direction by page, prompt stratum, engine and week |
| Uncertainty | Intervals where appropriate, raw counts and design limitations |
| Exceptions | Contamination, outages, model updates and excluded observations |
A small positive average driven by one page or one engine is not the same as a broad effect. Likewise, a null result may mean the treatment was ineffective, the effect was too small to detect, retrieval had not refreshed or the sample lacked sensitivity. State which interpretations remain plausible.
Use evidence grades for conclusions
I recommend assigning an evidence grade before deciding whether to roll out a change:
- Exploratory signal: convenience prompts, limited repetitions or no credible comparison.
- Directional evidence: pre-specified measures and repeated observations, but non-random controls or notable contamination.
- Moderate practical evidence: random or defensible staggered assignment, balanced baseline, stable coding and a consistent relative change.
- Inconclusive: mixed direction, insufficient observations, major platform disruption or poor implementation fidelity.
These are operational labels, not universal scientific standards. Their value lies in preventing a weak result from being promoted internally as proof.
Also separate mechanism claims from outcome claims. If a summary block is followed by more citations, you may have evidence that the package helped under the tested conditions. You usually do not know whether the system preferred its concision, extracted a particular sentence, responded to improved source support or refreshed for unrelated reasons.
Decide what happens after the test
A positive result does not require an immediate site-wide rollout. First inspect implementation cost, editorial quality, accessibility, maintenance burden and conventional search performance. A treatment that increases observed citations but makes pages repetitive or harder to maintain may be a poor standard.
Roll out in stages and retain monitoring prompts. If visibility later declines, do not assume the treatment stopped working; retrieval sources, competitors and answer products may have changed. Use a broader measurement framework such as measuring AI search visibility without misleading metrics.
For null or negative tests, verify treatment fidelity before abandoning the idea. Was the intended block rendered? Was it indexed? Did source links resolve? Did editors apply the specification consistently? A failed implementation is not a clean test of the editorial hypothesis.
FAQ
How long should an AI search visibility experiment run?
There is no universal duration. Run long enough to collect repeated observations across multiple retrieval cycles and ordinary day-to-day variation. Set the stopping rule in advance rather than ending when results look favourable.
Can schema markup be tested as the treatment?
Yes, provided eligibility and implementation can be verified. Google explains the general role of markup in its structured data documentation, but valid markup does not guarantee an AI citation or search appearance.
Should prompts be rewritten during the experiment?
Keep the primary sample fixed. Add emerging prompts to a labelled exploratory set, not the main outcome set, unless the protocol specifies a scheduled refresh.
Is a new citation proof that the edit worked?
No. It is an observation consistent with an effect. Stronger attribution requires a baseline, comparison group, repeated runs and evidence that unrelated changes do not better explain the difference.
Conclusion: make the claim no stronger than the design
Controlled AI search visibility experiments cannot remove every source of uncertainty. Commercial answer engines change without giving researchers control over models, retrieval indexes or personalisation. That limits causal confidence.
A disciplined protocol still improves decisions. Define one treatment, select prompts through a documented sampling process, assign credible comparisons, repeat observations, preserve raw outputs, track external events and analyse relative change. Then label the evidence honestly.
The practical standard is not perfect certainty. It is a result another practitioner could audit, repeat and challenge—and a conclusion that remains within what the experiment actually tested.
