AI Search Measurement Methodology

TL;DR — An AI-search result is a sample, not a ranking. Save the exact question, where it was asked, which model or product answered, when it ran, and what links appeared. Repeat the same test several times. Keep failures out of the denominator, and do not treat a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. visit, brand mention, citation, click, and conversion as the same event.

The record matters more than the score

An “AI visibilityLLM visibility (or AI visibility) is the aggregate measure of how often and how prominently a brand or page shows up in AI-generated answers — across AI Overviews, ChatGPT, Perplexity, Copilot, and Gemini. It's the AI-search analog of organic visibility, but it's driven by different signals. score” is difficult to interpret when you do not know what was tested. The same prompt can produce different answers across dates, models, countries, accounts, or repeated runs. A useful record therefore starts with the observation:

• the exact prompt and its version;
• the product, API, or official reporting surface;
• the requested and resolved model when known;
• whether web retrieval was on, off, or unavailable;
• the country, locale, device, and account state when relevant;
• the date, sample group, and repeat number;
• the answer or permitted evidence hash;
• every observed mention and citation;
• the passage apparently supporting each citation, when exposed;
• the extraction and methodology versions.

If a run failed, was refused, or did not expose the requested data, keep that record. Label it not evaluated rather than changing it to zero.

Five events that should remain separate

1. Fetched: a verified crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. or user-directed agent requested a URL.
2. Retrieved or selected: the page entered an observable retrieval set.
3. Mentioned: the answer named the entity.
4. Cited: the answer exposed a source URL.
5. Visited or converted: a person followed a link and completed a measured action.

A server log can prove the first event when the requester is properly verified. It normally cannot prove the remaining four. An answer capture can prove a visible mention or citation, but it cannot reveal every source the system retrieved or used internally.

Repeat before recommending a change

Run compatible prompts at least three times before applying a stability label; five is a better default. Show the result as an exact fraction—such as 3/5 eligible answers cited the page—rather than simply “60% visible.”

When a citation appears once and disappears twice, the correct initial interpretation is variability. It is not automatically an indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. problem or a penalty.

Try the AI Search Evidence Lab with its included example packet to see how eligible denominators, citation persistence, passage coverage, and fact conflicts are kept separate.

TL;DR — Use an append-only observation contract and a versioned prompt registry. Compare only compatible prompt/surface/retrieval/locale/method panels. Report binary outcomes with N and an uncertainty interval. Preserve raw evidence separately from derived classifiers and human corrections. Evaluate interventions with controls when possible, but keep even controlled observational results directional.

The observation envelope

The canonical observation should contain four layers:

Acquisition

Record the source type explicitly:

• consumer product;
• first-party model API;
• first-party API with web search;
• third-party provider;
• official webmaster report;
• verified server log;
• user upload.

A first-party API call with search is useful evidence about that API execution. It is not a direct measurement of a similarly named consumer product. Preserve that boundary in storage, UI labels, exports, and aggregations.

Experimental context

Store a stable prompt ID, immutable prompt version, prompt hash, surface, retrieval mode, locale, country, device, account state, sample group, and sample ordinal. Use a frozen benchmark panel for trends and a separate exploratory panel for discovering new prompt patterns.

When a prompt changes, create a new version. Do not edit historical observations to make them look compatible.

Evidence

Keep raw or hashed answer evidence, normalized cited and retrieved URLs, mention spans, citation positions when exposed, groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. queries when exposed, and supporting passages or passage hashes. Normalize tracking parameters and URL fragments without discarding the original URL.

For citations, a useful source record contains:

raw URL → normalized URL → observed canonical or redirect successor
        → answer citation position → supporting passage → observed HTTP status

That lineage prevents redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., parameters, syndicated copies, and migrations from inflating source counts.

Interpretation

Every extracted mention, sentiment, entity, fact, and loss reason should include:

• classifier or extractor version;
• observed, derived, inferred, or not-evaluated state;
• confidence and its reason;
• any later human correction.

A correction does not delete the original classification. It becomes a calibration fixture for evaluating the next classifier version.

Compatible comparisons

Before calculating a trend, require compatible values for:

• prompt ID and version;
• surface;
• retrieval mode;
• locale;
• methodology and extractor versions.

Annotate or break the series when the resolved model or checkpoint changes. Otherwise a provider update can look like a content-performance change.

For a binary result, report the numerator, eligible denominator, point estimate, and an uncertainty interval. Exclude refusals, provider failures, and unavailable evidence from the denominator, but continue showing their counts in run quality.

Public platform examples

These are examples of why source-specific contracts matter, not a request to force platforms into a shared ranking:

• Google’s public description of its Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. generative-AI report lists impressions, pages, countries, devices, and dates. Store those as the documented visibility fields; do not transform them into citations or answer rank. The report was announced as a limited rollout in June 2026. Official announcement
• Microsoft’s public Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. AI Performance description includes citation activity, cited pages, and sampled groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. queries. Microsoft explicitly says those citations do not indicate placement, authority, rank, or the role of a page in an answer. Official announcement

The adapter for each source should retain those semantics and mark undocumented fields unavailable.

Content-change experiments

Create an intervention record before judging an edit:

• hypothesis;
• affected and control URLs;
• prompt IDs;
• deployment time;
• observed recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. time;
• expected metric;
• before and after windows.

An uncontrolled before/after result is directional. A compatible control panel allows a difference-in-differences view, but it is still observational unless assignment and outside conditions justify stronger causal language.

Require the finding to persist across repeated runs before converting it into a content recommendation. The recommendation should link back to the observations that generated it.

Fact and contradiction monitoring

Maintain a human-reviewed fact ledger with entity, predicate, expected value, acceptable variants, primary source, validity dates, sensitivity, and last human verification. Compare observed answer claims with that ledger.

When models disagree, report conflict observed. Do not declare a winner until the values are checked against current evidence. Pricing, legal, medical, financial, and security facts deserve shorter verification windows and stronger review gates.

Crawler evidence

CrawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. controls differ by request purpose. OpenAI, Anthropic, and Perplexity document separate roles for model development, search/indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., or user-directed fetching. Verify identity using an operator-published IP list, documented reverse DNS, or another official mechanism where available; a user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string alone is not identity proof.

• OpenAI crawler documentation
• Anthropic crawler documentation
• Perplexity crawler documentation

Even a verified request proves only that request. Do not promote it into evidence of answer use, citation, traffic, or conversion.

Anti-patterns

• A universal score that hides disagreement between surfaces.
• A single prompt run labeled as share of voice.
• Treating unavailable evidence as zero.
• Comparing API output with a consumer product as if they were identical.
• Calling citations “rank.”
• Calling a 404 citation a hallucinationAn AI hallucination is when a large language model generates output that is confidently stated but factually wrong, made up, or unsupported by its source. It's a side effect of next-token prediction — not a bug that can be fully eliminated. before checking redirects and migrations.
• Rewriting historical prompts, observations, or corrections.
• Applying a classifier change retroactively without preserving its version.
• Claiming a content edit caused an increase without an appropriate experiment.
• Recommending llms.txt, special AI markup, or tiny chunks as universal Google requirements. Google’s current guidance says these are not required for its generative Search features. Official guidance