AI Search Measurement Methodology

TL;DR — An AI-search result is a sample, not a ranking. Save the exact question, where it was asked, which model or product answered, when it ran, and what links appeared. Repeat the same test several times. Keep failures out of the denominator, and do not treat a bot visit, brand mention, citation, click, and conversion as the same event.

The record matters more than the score

An “AI visibility score” is difficult to interpret when you do not know what was tested. The same prompt can produce different answers across dates, models, countries, accounts, or repeated runs. A useful record therefore starts with the observation:

• the exact prompt and its version;
• the product, API, or official reporting surface;
• the requested and resolved model when known;
• whether web retrieval was on, off, or unavailable;
• the country, locale, device, and account state when relevant;
• the date, sample group, and repeat number;
• the answer or permitted evidence hash;
• every observed mention and citation;
• the passage apparently supporting each citation, when exposed;
• the extraction and methodology versions.

If a run failed, was refused, or did not expose the requested data, keep that record. Label it not evaluated rather than changing it to zero.

Five events that should remain separate

1. Fetched: a verified crawler or user-directed agent requested a URL.
2. Retrieved or selected: the page entered an observable retrieval set.
3. Mentioned: the answer named the entity.
4. Cited: the answer exposed a source URL.
5. Visited or converted: a person followed a link and completed a measured action.

A server log can prove the first event when the requester is properly verified. It normally cannot prove the remaining four. An answer capture can prove a visible mention or citation, but it cannot reveal every source the system retrieved or used internally.

Repeat before recommending a change

Run compatible prompts at least three times before applying a stability label; five is a better default. Show the result as an exact fraction—such as 3/5 eligible answers cited the page—rather than simply “60% visible.”

When a citation appears once and disappears twice, the correct initial interpretation is variability. It is not automatically an indexing problem or a penalty.

Try the AI Search Evidence Lab with its included example packet to see how eligible denominators, citation persistence, passage coverage, and fact conflicts are kept separate.

TL;DR — Use an append-only observation contract and a versioned prompt registry. Compare only compatible prompt/surface/retrieval/locale/method panels. Report binary outcomes with N and an uncertainty interval. Preserve raw evidence separately from derived classifiers and human corrections. The broader NIST AI Risk Management Framework provides a complementary governance frame for that recordkeeping. Evaluate interventions with controls when possible, but keep even controlled observational results directional.

The observation envelope

The canonical observation should contain four layers:

Acquisition

Record the source type explicitly:

• consumer product;
• first-party model API;
• first-party API with web search;
• third-party provider;
• official webmaster report;
• verified server log;
• user upload.

A first-party API call with search is useful evidence about that API execution. It is not a direct measurement of a similarly named consumer product. Preserve that boundary in storage, UI labels, exports, and aggregations.

Experimental context

Store a stable prompt ID, immutable prompt version, prompt hash, surface, retrieval mode, locale, country, device, account state, sample group, and sample ordinal. Use a frozen benchmark panel for trends and a separate exploratory panel for discovering new prompt patterns.

When a prompt changes, create a new version. Do not edit historical observations to make them look compatible.

Evidence

Keep raw or hashed answer evidence, normalized cited and retrieved URLs, mention spans, citation positions when exposed, grounding queries when exposed, and supporting passages or passage hashes. Normalize tracking parameters and URL fragments without discarding the original URL.

For citations, a useful source record contains:

raw URL → normalized URL → observed canonical or redirect successor
        → answer citation position → supporting passage → observed HTTP status

That lineage prevents redirects, parameters, syndicated copies, and migrations from inflating source counts. Google’s canonicalization guidance describes the related URL-consolidation signals and their limits.

Interpretation

Every extracted mention, sentiment, entity, fact, and loss reason should include:

• classifier or extractor version;
• observed, derived, inferred, or not-evaluated state;
• confidence and its reason;
• any later human correction.

A correction does not delete the original classification. It becomes a calibration fixture for evaluating the next classifier version.

Compatible comparisons

Before calculating a trend, require compatible values for:

• prompt ID and version;
• surface;
• retrieval mode;
• locale;
• methodology and extractor versions.

Annotate or break the series when the resolved model or checkpoint changes. Otherwise a provider update can look like a content-performance change.

For a binary result, report the numerator, eligible denominator, point estimate, and an uncertainty interval. Exclude refusals, provider failures, and unavailable evidence from the denominator, but continue showing their counts in run quality.

Public platform examples

These are examples of why source-specific contracts matter, not a request to force platforms into a shared ranking:

• Google’s public description of its Search Console generative-AI report lists impressions, pages, countries, devices, and dates. Store those as the documented visibility fields; do not transform them into citations or answer rank. The report was announced as a limited rollout in June 2026. Official announcement
• Microsoft’s public Bing Webmaster Tools AI Performance description includes citation activity, cited pages, and sampled grounding queries. Microsoft explicitly says those citations do not indicate placement, authority, rank, or the role of a page in an answer. Official announcement

The adapter for each source should retain those semantics and mark undocumented fields unavailable.

Content-change experiments

Create an intervention record before judging an edit:

• hypothesis;
• affected and control URLs;
• prompt IDs;
• deployment time;
• observed recrawl time;
• expected metric;
• before and after windows.

An uncontrolled before/after result is directional. A compatible control panel allows a difference-in-differences view, but it is still observational unless assignment and outside conditions justify stronger causal language.

Require the finding to persist across repeated runs before converting it into a content recommendation. The recommendation should link back to the observations that generated it.

Fact and contradiction monitoring

Maintain a human-reviewed fact ledger with entity, predicate, expected value, acceptable variants, primary source, validity dates, sensitivity, and last human verification. Compare observed answer claims with that ledger.

When models disagree, report conflict observed. Do not declare a winner until the values are checked against current evidence. Pricing, legal, medical, financial, and security facts deserve shorter verification windows and stronger review gates.

Crawler evidence

Crawler controls differ by request purpose. OpenAI, Anthropic, and Perplexity document separate roles for model development, search/indexing, or user-directed fetching. Verify identity using an operator-published IP list, documented reverse DNS, or another official mechanism where available; a user-agent string alone is not identity proof.

• OpenAI crawler documentation
• Anthropic crawler documentation
• Perplexity crawler documentation

Even a verified request proves only that request. Do not promote it into evidence of answer use, citation, traffic, or conversion. OpenAI’s bot documentation, Anthropic’s crawler guidance, and Perplexity’s crawler documentation describe provider-specific identity and access boundaries.

Anti-patterns

• A universal score that hides disagreement between surfaces.
• A single prompt run labeled as share of voice.
• Treating unavailable evidence as zero.
• Comparing API output with a consumer product as if they were identical.
• Calling citations “rank.”
• Calling a 404 citation a hallucination before checking redirects and migrations.
• Rewriting historical prompts, observations, or corrections.
• Applying a classifier change retroactively without preserving its version.
• Claiming a content edit caused an increase without an appropriate experiment.
• Recommending llms.txt, special AI markup, or tiny chunks as universal Google requirements. Google’s current guidance says these are not required for its generative Search features. Official guidance

Changelog