AI Hallucination Monitoring

How to detect when AI tools fabricate or misstate facts about your brand — fabricated URLs, wrong pricing, invented quotes — and the workflow to catch and fix it.

First published: Jun 24, 2026 · Last updated: Jul 18, 2026 · Advanced
demand #6 in Measurement and Reporting#30 in AI Search#381 on the site
AIAuthored byAI v2

AI hallucination monitoring is catching AI tools when they make things up about your brand — wrong prices, invented quotes, fabricated URLs — and fixing it. AI bots read your content at massive scale (Claude crawls ~73,000 pages for every click it sends) but cite very little, so every wrong citation is amplified. My two Ahrefs field workflows catch different evidence: Best by Links → 404 finds fabricated URLs that accumulated links, while Web Analytics → Possible 404s filtered to the AI Search channel finds candidate URLs receiving AI-referred visits right now and lets you prioritize them by sessions. Validate every candidate before acting. Pair those passes with monthly manual prompt-testing and passive monitoring. When you find a hallucination, fix your own content first — platform feedback is slow and probabilistic.

The monitoring discipline here applies NIST’s general guidance to brand answers; it is not a NIST-prescribed SEO workflow. Evidence for this claim NIST recommends measuring and monitoring generative-AI risks, including confabulation, with empirically validated methods and documented limitations. Scope: General AI risk-management guidance, adapted here to brand-answer monitoring. Confidence: high · Verified: NIST: Generative AI Profile A sound test set should use task-specific examples and be rerun as systems change. Evidence for this claim OpenAI recommends task-specific evals, production data, and continuous evaluation rather than relying only on generic metrics or anecdotal tests. Scope: OpenAI evaluation guidance; the article's monitoring workflow is Patrick's editorial application of those principles. Confidence: high · Verified: OpenAI: Evaluation best practices

TL;DR — AI hallucinationAn AI hallucination is when a large language model generates output that is confidently stated but factually wrong, made up, or unsupported by its source. It's a side effect of next-token prediction — not a bug that can be fully eliminated. monitoring is the systematic detection, documentation, and remediation of AI tools fabricating or misstating facts about your brand. The scale framing: AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. read your content at enormous volume but cite almost none of it (Claude ~73,000:1 crawl-to-referral; ChatGPT ~3,700:1; Google ~14:1), so every wrong citation is amplified, not diluted. My signature technique is Ahrefs Best by Links + a 404 filter: hallucinated URLs that real people subsequently link to show up as backlinks pointing at pages that never existed. Cross-referencing those 404s against live AI tools is mandatory — not every 404-with-links is a hallucination. Layer manual prompt-testing and passive monitoring on top. When you confirm one, fix your source content first; Google provides no hallucination-correction channel, and platform feedback is slow and probabilistic.

Why this is a different problem than reputation monitoring

Traditional brand monitoring tracks what people say about you. Hallucination monitoring tracks what language models say — and the scale is in a different universe. In Cloudflare Radar data I analyzed, the Anthropic crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. was hitting sites at roughly a 73,000:1 ratio of crawls to clicks sent back. ClaudeBot ran around 23,951:1 in Q1 2026. ChatGPT sat near 3,700:1, Perplexity around 700:1. Google is down at about 14:1.

Read that the right way: AI systems are reading your content at massive scale and citing a vanishingly small fraction of it. The citations that do appear carry enormous implied authority — users trust them, and other writers cite them onward. So every hallucination inside a citation gets amplified, not diluted. That asymmetry is the whole reason to monitor citations specifically, not just crawl traffic.

The six kinds of brand hallucination to watch for

  1. Fabricated URLs — the AI cites yourdomain.com/features/ai-integration/, which has never existed.
  2. Wrong product features — it claims your tool does X when it does Y.
  3. Invented quotes — a statement attributed to your CEO that was never made.
  4. Incorrect pricing — your Pro plan listed at $49/mo when it’s $99/mo.
  5. Wrong company facts — founding year, team size, location, acquisition status.
  6. Competitive misattribution — a competitor’s differentiator credited to you, or yours credited to them.

Pricing is the one I’d triage first — it converts a hallucination straight into a mismatched customer expectation and lost revenue.

The Ahrefs Best by Links + 404 technique (my angle)

This is the part no competitor page covers properly, so I’ll go deep. The idea turns a passive backlink signal into an active hallucination detector.

Why backlinks accumulate on URLs that don’t exist. When an AI tool hallucinates a URL, it doesn’t invent gibberish — it generates a plausible path that matches your site’s URL patterns: /blog/seo-guide/, /features/reporting/, /pricing/enterprise/. Some users who get that response then publish content citing the URL as if it were real. That content gets indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Now there’s a real, indexed page linking to a non-existent URL on your domain — a genuine backlink pointing at a 404.

The supporting data is striking: in Ahrefs’ 16-million-URL study, AI assistants sent users to 404 pages404 Not Found is the HTTP client-error status code a server returns when it can't find the requested URL — RFC 9110 defines it as no current representation, or unwillingness to disclose one. A \"hard 404\" actually returns the 404 status; a \"soft 404\" returns a success code (like 200) for a page that's really gone. 404s are normal and expected: the fact that some URLs 404 doesn't affect your site's other, successful pages, and Google de-indexes 404'd URLs over time (probably retrying for some period, less and less often). 2.87x more often than Google Search, and ChatGPT’s all-cited-URL 404 rate hit 2.38%. In one analysis, about 20% of examined hallucinated URLs had at least one backlink — meaning a fifth of them had already been cited somewhere as real. One site foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. traffic to roughly 70 non-existent URLs in just three months of 404 logs.

The workflow (this is a UI process, no command line):

  1. Open Ahrefs Site Explorer for your domain.
  2. Go to Best by Links (under the Pages section in the left nav).
  3. Set the HTTP status filter to 404 Not Found404 Not Found is the HTTP client-error status code a server returns when it can't find the requested URL — RFC 9110 defines it as no current representation, or unwillingness to disclose one. A \"hard 404\" actually returns the 404 status; a \"soft 404\" returns a success code (like 200) for a page that's really gone. 404s are normal and expected: the fact that some URLs 404 doesn't affect your site's other, successful pages, and Google de-indexes 404'd URLs over time (probably retrying for some period, less and less often)..
  4. Sort by Referring Domains, most at the top.
  5. Export the list.
  6. For each 404 URL with meaningful referring domains — even one or two is notable for a page that doesn’t exist — run the confirmation steps below.

The confirmation steps (non-optional). This is where most write-ups stop short, and it’s the part that keeps you honest:

  • Check when the backlinks were created. New links forming around a URL that never existed is the signal. Old links usually mean a deleted or moved page, not a hallucination.
  • Read the anchor textAnchor text is the visible, clickable text of a hyperlink. It tells readers what they'll find on the other end and gives search engines context about the linked page. and surrounding context on the linking pages. Does it read like someone citing an AI-generated answer (“according to [Brand]‘s guide at…”)?
  • Cross-reference against a live AI tool. Ask it directly: “What is [your brand]‘s page on [topic]?” and see whether it generates that exact URL. If it does, you’ve confirmed the source.

The accuracy spine, stated plainly: not every 404 with backlinks is a hallucination. Pages get deleted or moved without redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't.. URLs change in migrations. External sites typo your URLs. A 404 only becomes a hallucination candidate when (a) the page never existed, (b) new links are appearing rather than old ones persisting, and (c) the AI cross-reference confirms a tool is generating it. Skip step (c) and you’ll chase normal broken links as if they were AI ghosts.

And the limit of the technique: Best by Links + 404 only catches hallucinated URLs that someone subsequently linked to. A hallucination that no one ever linked to leaves no backlink trail — it won’t show up here. This method is one net, not the whole ocean. Pair it with manual prompt-testing and 404 log monitoring to catch the rest.

The traffic companion: Possible 404 × AI Search

Best by Links answers, “Which missing URLs accumulated links?” My second signature technique answers a more urgent question: which missing URLs are receiving visits from AI assistants right now? Ahrefs documented the underlying signal in its study of hallucinated links: Web Analytics can identify AI-assistant referrers and flag a possible 404 when the visited page titleThe title tag is the HTML title element in a page's head that specifies the document's title. It's the primary source for the SERP title link and a confirmed light ranking factor — but since August 2021 Google doesn't always show it verbatim. contains language such as “404” or “not found.” That is traffic evidence, not a crawl-status verdict, so the word possible matters.

The workflow:

  1. Open Ahrefs Web Analytics for the site and go to its pages reporting.
  2. Apply the Possible 404 page filter.
  3. Filter the channel to AI SearchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. so ordinary search, direct, referral, and internal visits do not blur the signal.
  4. Sort the remaining URLs by sessions, highest first. Preserve the URL, title/status evidence, sessions, channel, and date-range context in the export or case log.
  5. Request every candidate directly and confirm its actual response, redirect path, and history. A custom 404 template can return 200; a legitimate live page can contain “not found” in its title; and an old URL may have existed long before an AI assistant linked to it.

This is the traffic-based companion to Best by Links, not a replacement for it. Links reveal citations that persisted long enough to be published elsewhere; sessions reveal AI-referred clicks happening in the selected window. A URL with ten current AI Search sessions deserves review before a zero-session URL with one old backlink, but sessions do not prove hallucination. Confirm that the URL never existed, inspect the referring AI source when available, and reproduce the citation before classifying it.

For a confirmed candidate, preserve the session count in the case log and use it to set response priority. Then make the same relevance decision as with the backlink workflow: create the genuinely missing resource, redirect only to a true equivalent, or leave a correct 404/410. My 404 audit guide404 Not Found is the HTTP client-error status code a server returns when it can't find the requested URL — RFC 9110 defines it as no current representation, or unwillingness to disclose one. A \"hard 404\" actually returns the 404 status; a \"soft 404\" returns a success code (like 200) for a page that's really gone. 404s are normal and expected: the fact that some URLs 404 doesn't affect your site's other, successful pages, and Google de-indexes 404'd URLs over time (probably retrying for some period, less and less often). explains the response-code decision; the redirects hubA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. explains how to avoid turning this discovery pass into a homepage-dump problem.

What to do with a confirmed hallucinated URL:

  • Meaningful backlinks or traffic — treat it as an accidental content-gap signal. Either build the page people clearly expect, or 301 redirectA 301 redirect is the HTTP status code for a permanent move: it tells browsers and search engines a URL has moved for good, and it's the strongest signal for consolidating a page's ranking signals onto the new URL. Google says permanent redirects don't cause a loss in PageRank. to the most relevant existing page to recover the link equity.
  • Moderate backlinks, no traffic — 301 to the closest topically relevant page.
  • Trivial volume — monitor; don’t over-invest.
  • Every case — update your structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding. and on-page facts so AI tools have the correct URL structureURL structure is how the parts of a web address — scheme, domain, path, query string, and fragment — are organized and formatted. It mostly affects crawling, usability, and how engines understand a page, not rankings directly. to learn from.

Manual prompt-testing

The Ahrefs method catches fabricated URLs. Manual testing catches everything else — wrong prices, invented quotes, bad facts. Run these across ChatGPT, Gemini / AI Overviews, Copilot, Perplexity, Claude, and Grok:

  • “What does [Brand] do?”
  • “What are [Brand]‘s pricing plans?”
  • “Who founded [Brand] and when?”
  • “What is [Brand]‘s URL for [feature/topic]?”
  • “What are [Brand]‘s key features compared to [Competitor]?”
  • “What has [CEO Name] said about [topic]?”

Log date, tool, model version, the exact prompt, the verbatim response, the specific inaccuracies, and any URLs cited. A spreadsheet is fine. Frequency: monthly minimum, weekly if you’re in a fast-moving category or your product changes often. And note the counterintuitive part — smaller and newer brands need this more, not less. Big brands have dense, authoritative training data; a brand that launched in the last two years is barely represented, so the model’s “facts” about it are closer to lucky guesses.

TIP Check sampled answers against facts you already know

Use this as a repeatable fact-check, not as a universal visibility score. The pictured run evaluates one simulated provider response and explicitly leaves the other providers unevaluated.

Record a known brand fact and compare it with a sampled answer using my free AI Brand Visibility Checker Free

  1. Define the exact fact to verify, such as a founding year, price, feature, or canonical product name.
  2. Run the same fact and prompt across provider-specific samples rather than treating an unevaluated provider as zero visibility.
  3. Log the provider, model, retrieval mode, date, and evidence before deciding whether the answer is wrong or merely incomplete.
A missing known fact is a review cue, not proof of a hallucination by itself. Confirm the intended answer, provider, retrieval state, and source evidence.

The completed AI Brand Visibility Checker sample uses a simulated Llama response with retrieval off. It mentions the fictional Acme Analytics brand, but the known-fact check marks Founded in 2018 as missing because that supplied fact was not clearly found in the response. Mistral, ChatGPT Search, Gemini, Claude, and Perplexity are each labeled not evaluated, with a warning not to infer a zero mention or citation rate. The tool recommends confirming the intended state before making a change.

Passive and automated monitoring

Free / low-cost:

  • Google Alerts on "[Brand Name]" — surfaces new web content mentioning you, including AI-generated articles that may repeat a hallucination.
  • Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — the AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. filter shows which of your pages Google is citing. Pages it’s not citing are pages it’s less likely to treat as an accurate source.
  • Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — AI Performance ReportThe Google Search Console report that shows how your site actually performed in Google Search, built from real impressions and clicks. It reports four metrics — clicks, impressions, average CTR, and average position — and keeps the most recent 16 months of data. — the Copilot equivalent.
  • 404 log / crawl monitoring — catch hallucinated URLs early, before they accumulate links.

Paid tooling (each takes a different cut at the problem):

  • Ahrefs Brand Radar — scheduled custom-prompt tracking; mentions, accuracy, and sentiment over time.
  • Visiblie — monitors brand mentions across up to eight LLMs with built-in hallucination detection and accuracy tracking.
  • Dageno — hallucination correction tools plus AI visibility monitoring and page-level GEOGenerative Engine Optimization — visibility inside AI answer engines. content audits.
  • Brand Armor AI — dedicated hallucination monitoring with prompt-level accuracy scoring.
  • Relixir — real-time hallucination alerts.
  • Scrunch — sentiment monitoring (catches negative or skewed framings even when they aren’t strict factual errors).

My ongoing Ahrefs loop: set up Brand Radar prompts that test known-accurate facts (pricing, features, key URLs); review new referring domains weekly, watching for any that resolve to a 404; and schedule a monthly Best by Links → 404 export to catch newly hallucinated URLs as they pick up links.

What to do when you confirm a hallucination

  1. Document it. Log the claim, the tool, the date, a screenshot. If the same error shows up across multiple tools, note it — that often points to a shared training-data source.
  2. Fix the source content. This is the fastest, most reliable lever. Update every page that mentions the wrong fact, and publish a clear, authoritative, schema-marked page stating the correct one. You’re trying to flood the retrieval and training pipeline with accurate signal.
  3. Submit platform feedback (details in the Official Docs tab). Real, but slow and probabilistic — no guaranteed timeline.
  4. Reinforce with structured data — Organization, Product/Offer, Person, and FAQ schema that answers the commonly hallucinated questions directly.
  5. Strengthen third-party signals — Wikipedia/Wikidata, Crunchbase, LinkedIn, industry directories, and the Reddit threads where your brand gets discussed. AI tools weight these heavily.

The order matters. Fixing your own content is faster than waiting on the platforms — and for Google specifically, it’s the only lever, because there is no publisher-facing hallucination-correction channel at all (see Official Docs).

Where this sits

This is the how-to-catch-them companion to AI HallucinationsAn AI hallucination is when a large language model generates output that is confidently stated but factually wrong, made up, or unsupported by its source. It's a side effect of next-token prediction — not a bug that can be fully eliminated. (the what and why) and a measurement discipline alongside LLM visibilityLLM visibility (or AI visibility) is the aggregate measure of how often and how prominently a brand or page shows up in AI-generated answers — across AI Overviews, ChatGPT, Perplexity, Copilot, and Gemini. It's the AI-search analog of organic visibility, but it's driven by different signals. and AI share of voiceAI Share of Voice (SoV) measures how often and how prominently a brand appears in AI-generated responses relative to competitors, across a defined pool of relevant prompts. It's a visibility signal, not a traffic or revenue metric.. Where those measure how present you are in AI answers, hallucination monitoring measures how accurate that presence is.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.