Multimodal Search

TL;DR — Multimodal searchMultimodal search lets you query and get results across more than one modality — text, images, video, and audio together. Instead of typing words, you can point your camera, circle something on screen, or combine an image with a question, and the system understands them jointly. means you don’t have to type to search. You can point your camera at something (Google Lens), circle a thing on your screen (Circle to Search), or speak a question — and combine those with words. The engine understands the picture, the voice, and the text together, then answers. It’s the same “match by meaning” idea as modern search, just no longer limited to written words.

What multimodal search is

Multimodal systems process or relate more than one modality, such as text and images. Evidence for this claim CLIP learns joint image-and-text representations from paired internet data and supports zero-shot image classification. Scope: CLIP research results; multimodal search products may use different models, training data, and ranking pipelines. Confidence: high · Verified: 2026-07-14 Radford et al.: Learning Transferable Visual Models Product capabilities vary, so image, video, audio, and text support should be verified per system. Evidence for this claim Google Multisearch lets users combine an image with text to refine a search. Scope: Google's documented consumer search feature; it does not define all multimodal retrieval systems. Confidence: high · Verified: 2026-07-14 Google: Multisearch

For most of search’s history there was one way in: you typed words, and you got a list of links. Multimodal search breaks that open. A modality is just a type of input or output — text, an image, video, audio. Multimodal search means the engine can take more than one of those at once and make sense of them jointly.

Some everyday examples you’ve probably already used:

• Point your camera at a plant, a landmark, or a pair of shoes and ask “what is this?” — that’s Google Lens.
• Circle something on your phone screen — a jacket in a photo, a word in an article — to search it without leaving the app. That’s Circle to Search.
• Talk to your phone instead of typing — that’s voice search.
• Combine an image and a question: show a photo of a chair and ask “where can I buy this in blue?” The picture and the words are one query.

Why it works: matching by meaning

The trick under the hood is that modern systems don’t treat a picture and a sentence as totally different things. They translate both into the same internal “language” — a list of numbers that captures meaning (see embeddingsEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords.). A photo of a golden retriever and the words “golden retriever” end up close together in that space, so the engine can match them even though one is pixels and the other is letters. That’s the same meaning-based matching behind regular modern search (semantic searchSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it.) — just extended so it works across images, voice, and video too.

What it means for you

Here’s the honest version, because there’s a lot of hype here: there’s no secret “multimodal ranking factor” you flip on. What actually helps is the stuff good sites already do, just with a clearer reason:

• Describe your images. Real alt textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search. and sensible filenames tell the engine what a picture is.
• Give audio and video real text. Transcripts and captions let the engine understand — and quote — what’s said.
• Answer the obvious question. If someone photographs your product, the query they’ll ask is “what is this / where do I get it / how much.” Make that easy to find on the page.

Want the real mechanics — how MUM and AI Mode fit in, what Google has actually confirmed vs. what’s just theory, and the full playbook? Switch to the Advanced tab.

TL;DR — Multimodal searchMultimodal search lets you query and get results across more than one modality — text, images, video, and audio together. Instead of typing words, you can point your camera, circle something on screen, or combine an image with a question, and the system understands them jointly. is querying and retrieving across text, image, video, and audio in a single interaction. The enabler is a shared embeddingEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. space: different modalities get encoded into the same vector space so they can be matched by meaning — the same retrieval idea as semantic searchSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it., generalized past text. Google’s public arc runs from voice and image search to MUM (2021, multimodal, “1,000 times more powerful than BERT,” used for narrow high-complexity cases) to Google Lens, Circle to Search, and Gemini-powered AI Mode, where a photo plus a follow-up is one query that fans out into a synthesized answer. What’s officially confirmed is mostly about understanding the query across modalities; claims that Google ranks your individual images or video frames as a distinct “multimodal signal” are industry theory. The durable playbook: descriptive alt text/filenames, real transcripts/captions, clean structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding., and content that answers the implied question behind a visual or spoken query.

What “multimodal” actually means

Shared representation research shows one approach to cross-modal retrieval, not a universal production architecture. Evidence for this claim CLIP learns joint image-and-text representations from paired internet data and supports zero-shot image classification. Scope: CLIP research results; multimodal search products may use different models, training data, and ranking pipelines. Confidence: high · Verified: 2026-07-14 Radford et al.: Learning Transferable Visual Models Search-product descriptions document features but not complete ranking mechanics. Evidence for this claim Google Multisearch lets users combine an image with text to refine a search. Scope: Google's documented consumer search feature; it does not define all multimodal retrieval systems. Confidence: high · Verified: 2026-07-14 Google: Multisearch

A modality is a channel of information — written text, an image, spoken audio, video. A unimodal system handles one; a multimodal system takes more than one as input, or produces more than one as output, and — crucially — reasons about them jointly rather than bolting separate pipelines together.

Multimodal search is broader than “image search,” and the difference matters:

• Image search finds images (you want pictures).
• Multimodal search uses an image (or voice, or video) as part of a query that can return any kind of result — a shopping listing, a how-to, a definition, a map pin. The camera is an input device, not the thing you’re looking for.

So “point your camera at a broken part and ask how to fix it” is multimodal search; “find me photos of that part” is image search. Google Lens does both, which is part of why the line blurs.

It also helps to keep three separate questions apart, because a single product description can quietly blur them: what modality can the query accept (text only, or text plus image, audio, video)? What modality is the retrieved candidate content in (a text page, an image, a video)? And what modality is the generated output (a list of links, an image grid, a synthesized text answer)? A feature that accepts an image input doesn’t automatically retrieve image results or produce an image output — a photo of a broken part can retrieve a text page and generate a text answer. Confirming one of these three doesn’t confirm the other two; check each one against what the provider has actually documented for that specific feature.

The technical enabler: one shared space

The reason a picture and a phrase can be compared at all is shared representation. Encoder models map text, images, audio, and video into the same high-dimensional embedding space, where semantic similarity becomes geometric closeness. A photo of a plant and the string “how do I care for this plant” land near each other, so a nearest-neighbor lookup can connect them.

This is exactly the machinery behind semantic searchSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it. and the retrieval leg of RAGRAG is the retrieve-then-generate pattern behind AI search: the system retrieves relevant passages from an external index at query time, injects them into the model's context, and generates an answer grounded in those sources — without changing the model's weights., generalized beyond text, and it breaks into distinct stages — each one a separate capability, so evidence that a system is good at one stage doesn’t confirm it’s good at the others:

1. Input understanding. The query (image + text, say, or a spoken question) is parsed into whatever representation the next stage needs.
2. Embedding. That representation, and every candidate piece of content, is encoded into the shared vector space described above.
3. Retrieval. Vector searchVector search finds content by comparing the meaning of a query against stored content as numerical vectors, retrieving the closest ones in a high-dimensional embedding space. At scale it uses approximate nearest neighbor (ANN) algorithms — not exact comparison — to search billions of vectors in milliseconds. finds the closest matches by geometric distance.
4. RerankingReranking is the second stage of a retrieval pipeline: after a cheap, broad first pass pulls a candidate set of documents or passages, a slower, more precise model re-scores and reorders that shortlist by true relevance before the results are served or handed to an LLM.. A separate stage reorders the retrieved candidates using signals beyond raw embedding distance.
5. Generation. In AI surfaces, the top candidates are grounded into a synthesized answer.

If you understand embeddingsEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords., chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking., vector searchVector search finds content by comparing the meaning of a query against stored content as numerical vectors, retrieving the closest ones in a high-dimensional embedding space. At scale it uses approximate nearest neighbor (ANN) algorithms — not exact comparison — to search billions of vectors in milliseconds., and groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it., you already understand multimodal search’s plumbing — the only new part is that the encoders speak more than one modality.

Named research systems, not a confirmed production blueprint. The joint-embedding idea isn’t hypothetical — OpenAI’s CLIP and Google’s ALIGN both trained image and text encoders into one shared space using image-text pairs, and Meta’s ImageBind extended the approach to six modalities (image, text, audio, depth, thermal, and motion) in a single space. Those are named, published research architectures with their own datasets and evaluations — useful for understanding how a shared space can be built, not proof that any specific commercial search engine’s production system uses that exact design. A provider’s output behavior alone doesn’t reveal whether it uses a shared embedding space, a different fusion method, or how it weights each modality — that’s internal to a system nobody outside the provider can inspect.

Google’s public arc

Google’s confirmed steps toward multimodal search, roughly in order:

• Voice and image search (2010s). Spoken queries and reverse-image lookups were the first mainstream non-text inputs.
• MUM — Multitask Unified Model (2021). Google’s public marker for the multimodal shift. Google described MUM as multimodal — able to “understand information across text and images” — and “1,000 times more powerful than BERT,” trained across 75 languages, and able to both understand and generate language. The important caveat, straight from the semantic-search history: MUM was never Google’s general ranking engine. It was applied to narrow, high-complexity cases (complex multi-step questions, some featured snippets, shopping), and it did not “replace” BERT.
• Google Lens. Camera-as-query at scale — identify objects, translate text in an image, shop what you see, solve a problem you photograph.
• Circle to Search. Circle, highlight, or tap anything on your Android screen to search it in place, without switching apps.
• AI Mode (Gemini-era). The current frontier: a multimodal query (an image plus a spoken or typed follow-up) becomes a single interaction that fans out into many sub-queries and returns a synthesized, cited answer. This is where multimodal input meets the agentic, RAG-style retrieval described in this cluster.

Microsoft has its own line — Bing Visual Search and Copilot Vision — that does the analogous thing on Bing’s side, so multimodal search isn’t a Google-only phenomenon.

Where it can fail before retrieval even starts

Every modality other than clean text has to pass through a conversion step before the shared-space matching described above can happen, and that step can introduce errors the retrieval stage never sees coming. Provider technical documentation on multimodal models describes exactly this class of limitation:

• OCR errors — text embedded in an image (a sign, a screenshot, a label) can be misread, especially at low resolution or in an unfamiliar font.
• Speech recognition errors — a mistranscribed word in audio or video propagates into whatever gets embedded and retrieved.
• Frame sampling — video isn’t understood frame-by-frame; systems sample a subset of frames, so content that only appears between sampled frames can be missed entirely.
• Cropping and resolution — a system working from a low-resolution or tightly cropped image has less to work with than the original scene.
• Language — accuracy in one language doesn’t establish accuracy in another; OCR and speech recognition performance both vary by language.
• Missing context — an image or clip stripped of its surrounding page context (caption, alt textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search., transcript) gives the system less to anchor understanding to.

None of this is unique to any one provider, and it’s a reason to treat “the model can see/hear X” claims cautiously — a capability that works on a clean, well-lit, high-resolution test example doesn’t establish it works on the messier real-world version of the same input.

What’s confirmed vs. what’s theory

This is where careful writing matters, because the topic attracts overreach.

Reasonably confirmed:

• Search engines accept and understand multimodal queries — image, voice, and screen-circling inputs are real, shipping products.
• The retrieval behind AI answers is meaning-based (embeddings / semantic retrieval) and, in AI Mode, grounded in the core Search index — there’s no separate “AI index.”
• Google explicitly says its AI features rely on core Search ranking and quality systems, so the crawl → index → retrieve chain still governs whether your content can show up at all.

Industry theory — flag it as such:

• That Google ranks your individual images or video frames as a distinct “multimodal ranking signal” you can optimize in isolation. Google has not published a “multimodal ranking factor.” How deeply page-level visual/audio content is understood frame-by-frame at web scale is only partly documented.
• Precise claims about how much weight a multimodal query gives to the image vs. the accompanying text. Treat specific ratios as speculation.

The safe reading: optimize for being understood and retrievable across modalities, not for a mechanism Google hasn’t confirmed.

What this means for SEO

Strip the hype and the playbook is concrete and familiar:

• Make images legible to machines. Descriptive alt textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search., meaningful filenames, and (where relevant) image structured data. Alt text is how a search engine knows what a picture depicts without “seeing” it — and it still helps even as visual understanding improves.
• Give audio and video real text. Transcripts, captions, and clear on-page context turn spoken content into something retrievable and quotable. A video with no transcript is far harder to ground into an answer.
• Structured data where it fits. Product, ImageObject, VideoObject, and Recipe-type markup give machines unambiguous facts to attach to a visual or spoken query.
• Answer the implied question. A visual query carries an intent — “what is this,” “where do I buy it,” “how do I fix it.” Content that answers that intent directly, in a self-contained passage, is what gets retrieved. This is the same passage-level clarity that passage rankingPassage ranking is a Google AI system that scores individual sections ('passages') of a page so a single page can earn multiple relevance scores for different queries. Google still indexes whole pages — only the ranking changed. and RAG already reward.
• The prerequisites don’t change. Multimodal answers still retrieve from the core index, so being crawlable and indexed comes first — the same chain covered in crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it., and RAG.

None of this is a guarantee. Google’s own image and video documentation frames alt text, transcripts, and structured data as eligibility and understanding aids — things that help a search engine discover, understand, and correctly attach your content to a query — not as inputs to a documented ranking formula. Doing all of the above makes your content retrievable and citable; it doesn’t guarantee retrieval, citation, ranking position, or that any generated answer describes it accurately.

The one-sentence version: multimodal search widened the front door, but it didn’t change what’s behind it. Be findable, be clear about what your images and media depict, and answer the question the query implies.

AI summary

A condensed take on the Advanced version:

• Multimodal searchMultimodal search lets you query and get results across more than one modality — text, images, video, and audio together. Instead of typing words, you can point your camera, circle something on screen, or combine an image with a question, and the system understands them jointly. = querying/retrieving across text, image, video, and audio in one interaction. Broader than image search: it uses a picture/voice/video as input to return any kind of result, not just pictures.
• Three separate dimensions. What modality the query accepts, what modality the retrieved content is in, and what modality the output is are three different questions — confirming one doesn’t confirm the others.
• Enabler = a shared embeddingEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. space, built through five distinct stages (input understanding, embedding, retrieval, rerankingReranking is the second stage of a retrieval pipeline: after a cheap, broad first pass pulls a candidate set of documents or passages, a slower, more precise model re-scores and reorders that shortlist by true relevance before the results are served or handed to an LLM., generation). Named research systems (CLIP, ALIGN, ImageBind) demonstrate joint embedding spaces exist and work — they’re not proof of any specific provider’s production architecture, which output behavior alone can’t reveal.
• Google’s arc: voice + image search → MUM (2021, multimodal, “1,000 times more powerful than BERTSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it.,” 75 languages — but a narrow high-complexity tool, never the general ranking engine) → Google Lens → Circle to Search → Gemini-era AI Mode, where an image + follow-up fans out into a grounded, cited answer. Microsoft parallels: Bing Visual Search and Copilot Vision.
• Where it fails before retrieval: OCR misreads, speech-recognition errors, video frame sampling gaps, cropping/resolution limits, language variance, and missing surrounding context can all corrupt understanding before matching even starts.
• Confirmed: multimodal queries are real; AI retrieval is meaning-based and grounded in the core Search indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. (no separate AI index); crawl → index → retrieve still governs eligibility.
• Industry theory (flag it): a distinct “multimodal ranking signal” over your individual images/frames, and precise image-vs-text weighting. Google hasn’t confirmed these.
• SEO playbook: descriptive alt textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search. + filenames, real transcripts/captions, Product/ImageObject/VideoObject structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding., and self-contained content that answers the implied intent of a visual/spoken query. Prerequisites (crawlable, indexed) unchanged — and none of it guarantees retrieval, ranking, or citation; Google documents these as eligibility/understanding aids, not ranking inputs.

Official documentation

Primary-source material from the search engines on multimodal surfaces and the systems behind them.

Google

• MUM: A new AI milestone for understanding information — the 2021 announcement of the multimodal, multilingual Multitask Unified Model.
• How AI powers great search results — how RankBrainSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it., neural matching, BERT, and MUM fit together in Google’s own words.
• Google’s Guide to Optimizing for Generative AI Features — AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. / AI Mode rely on core Search ranking; groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it./RAGRAG is the retrieve-then-generate pattern behind AI search: the system retrieves relevant passages from an external index at query time, injects them into the model's context, and generates an answer grounded in those sources — without changing the model's weights. and query fan-outQuery fan-out is the technique where an AI search system breaks a single user question into multiple related sub-queries, runs those searches concurrently, and synthesizes the retrieved results into one answer. Google confirms AI Overviews and AI Mode 'may use a query fan-out technique' issuing multiple related searches across subtopics..
• AI Mode and AI Overviews updates — the evolving multimodal AI Mode experience.
• Visual search with Lens on Shopping — camera-as-query applied to shopping.
• Image SEO best practices — descriptive alt textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search., filenames, and image structured dataImageObject is the schema.org type that describes an image as structured data. It's rarely used alone — its main job is as the value of an image or logo property nested inside another type (Product.image, Article.image, Recipe.image, Organization.logo). You upgrade a plain image URL to a full ImageObject when you want to attach metadata: who owns the image, how it can be licensed, its caption, or its dimensions. Adding the license property is what makes an image eligible for Google's Licensable badge. (the machine-legibility fundamentals).
• Video SEO best practices — VideoObject, transcripts, and how Google understands video.

Microsoft / Bing

• Bing Visual Search — Bing’s image-as-query surface.
• Copilot Vision — Microsoft’s multimodal assistant that can “see” and reason about what’s on screen, available through the Copilot app on Windows.

Quotes from the source

On-the-record statements from Google on the multimodal shift. Deep links jump to the quoted passage where the page supports it.

Google — MUM is multimodal

• MUM is “multimodal, so it understands information across text and images and, in the future, can expand to more modalities like video and audio.” — Pandu Nayak, Google, on the MUM announcement. Read the announcement
• MUM is “1,000 times more powerful than BERTSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it.” and, unlike BERT, can both understand and generate language across 75 languages. Read the announcement

Google — AI features run on the core Search indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.

• “Our generative AI features on Google Search are rooted in our core Search ranking and quality systems.” — Google Search Central, AI optimization guide. (This is why the crawl → index → retrieve chain still governs multimodal AI eligibility.) Read the guide

Which multimodal surface am I actually optimizing for?

There isn’t a single “multimodal SEO” knob — the useful work depends on how people reach your content. Walk the branch that matches.

1. Are people photographing physical products to find or buy them? → You’re in Google Lens / visual shopping territory. Prioritize clean product photography, Product + ImageObject structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding., descriptive alt textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search. and filenames, and an accurate merchant/product feed. The query’s intent is “what is this / where do I buy it / how much.”

2. Do people circle or screenshot things inside your content to learn more? → Think Circle to Search / on-screen queries. The lever is on-page clarity: label diagrams and images, keep captions descriptive, and make sure the surrounding text answers the obvious follow-up so a circled term resolves to your explanation.

3. Is your content primarily spoken or video? → You’re optimizing for voice queries and video understanding. Publish real transcripts and captions, add VideoObject markup, and put a plain-text summary near the media. A search engine can only retrieve and quote what it can read.

4. Do you want to show up in AI Mode / AI OverviewAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index. answers for visual or spoken queries? → This is groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. / RAGRAG is the retrieve-then-generate pattern behind AI search: the system retrieves relevant passages from an external index at query time, injects them into the model's context, and generates an answer grounded in those sources — without changing the model's weights., not a separate multimodal channel. Be crawlable and indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. first, then write self-contained passages that answer the implied sub-questions a fanned-out multimodal query would ask (see passage rankingPassage ranking is a Google AI system that scores individual sections ('passages') of a page so a single page can earn multiple relevance scores for different queries. Google still indexes whole pages — only the ranking changed., RAG, groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it.).

5. Are you being told to “optimize for the multimodal ranking algorithm”? → Stop. Google hasn’t published a distinct multimodal ranking factor over your images or frames — that’s industry theory. Do the confirmed fundamentals above and don’t chase a mechanism that hasn’t been confirmed.

The mental models

1. A modality is just a channel. Text, image, audio, video. Unimodal = one channel; multimodal = more than one, reasoned about together. The word sounds exotic; the idea isn’t.

2. Multimodal searchMultimodal search lets you query and get results across more than one modality — text, images, video, and audio together. Instead of typing words, you can point your camera, circle something on screen, or combine an image with a question, and the system understands them jointly. ≠ image search. Image search returns pictures. Multimodal search uses a picture (or voice, or video) as input to return any kind of result. The camera is an input device, not the destination.

3. One shared space is the whole trick. Different modalities get encoded into the same embeddingEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. space, so a photo and a phrase can be compared by meaning. If you know semantic searchSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it. and vector searchVector search finds content by comparing the meaning of a query against stored content as numerical vectors, retrieving the closest ones in a high-dimensional embedding space. At scale it uses approximate nearest neighbor (ANN) algorithms — not exact comparison — to search billions of vectors in milliseconds., you know multimodal search — the encoders just speak more languages.

4. Confirmed query vs. theorized ranking. Multimodal queries are confirmed and shipping. A distinct multimodal ranking signal over your media is not. Optimize for being understood and retrievable, not for an unconfirmed mechanism.

5. The prerequisite chain is unchanged. Multimodal AI answers retrieve from the core indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Crawl → index → retrieve still gates everything, so fundamentals (crawlable, indexed, clear) come before anything “multimodal-specific.”

6. Machine legibility is the lever. You can’t hand the engine pixels and hope. Alt textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search., filenames, transcripts, captions, and structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding. are how a visual or spoken thing becomes something a search system can match and cite.

Multimodal search — cheat sheet

What it is in one line Querying and retrieving across text, image, video, and audio in one interaction — enabled by a shared embeddingEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. space, the same meaning-based retrieval as semantic search, extended past text.

The surfaces

Multimodal searchMultimodal search lets you query and get results across more than one modality — text, images, video, and audio together. Instead of typing words, you can point your camera, circle something on screen, or combine an image with a question, and the system understands them jointly. vs. image search

Confirmed vs. theory

The confirmed playbook

• Descriptive alt textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search. + meaningful filenames.
• Real transcripts and captions for audio/video.
• Product / ImageObject / VideoObject structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding..
• Self-contained passages that answer the implied query intentSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it..
• Crawlable + indexed first — no separate AI index.

Multimodal-readiness checklist

A pass to confirm your content can be understood and retrieved across modalities:

• Every meaningful image has descriptive alt textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search. (not keyword-stuffed, not empty on content images).
• Image filenames describe the subject (blue-wingback-chair.jpg, not IMG_4821.jpg).
• Product/visual pages carry appropriate structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding. (Product, ImageObject, Recipe, etc.).
• Videos have a transcript and/or captions, plus VideoObject markup and an on-page text summary.
• Audio/podcast content ships with a readable transcript.
• Diagrams and screenshots have descriptive captions and surrounding text that answers the obvious follow-up question.
• Pages that a visual query would land on directly answer the implied intent (“what is this / where to buy / how to fix”) in a self-contained passage.
• Images and media aren’t blocked from crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., and the pages hosting them are indexable (the retrieve chain is intact).
• You’re not relying on an unconfirmed “multimodal ranking factor” — the plan rests on the confirmed fundamentals above.

Inventory image context in the browser

Run this in the Chrome DevTools Console on a representative page. It lists each image’s source, dimensions, alternative textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search., and nearest heading:

console.table([...document.images].map(img => ({ src: img.currentSrc || img.src, width: img.naturalWidth, height: img.naturalHeight, alt: img.getAttribute('alt'), heading: img.closest('section, article, main')?.querySelector('h1,h2,h3')?.textContent?.trim() || '' })));

Missing alt and a weak surrounding section are review flags, not automatic SEO failures. Decorative images can correctly use empty alternative text.

Find images that require interaction to acquire a URL

Run this before clicking galleries or carousels:

const urls = new Set([...document.images].map(i => i.currentSrc || i.src));
console.table([...document.querySelectorAll('[data-src], [data-lazy-src]')].map(el => ({ pending: el.dataset.src || el.dataset.lazySrc, alreadyRendered: urls.has(el.dataset.src || el.dataset.lazySrc) })));

A pending image is not necessarily inaccessible, but the result identifies assets that need a rendered-page crawl and interaction review.

Verify image assets and context after publishing

Test to run: Fetch the page with a rendered crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and export image URLs, status codes, alternative textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search., and referring page headings. Expected result: Important images return successfully and appear in rendered HTML with accurate page context. Failure interpretation: The asset is broken, loaded only after an unsupported interaction, or separated from the text that explains it. Monitoring window: Immediate after deployment. Rollback trigger: The release removes or breaks important product, instructional, or primary content images.

Verify structured image associations

Test to run: Inspect any relevant structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding. and its referenced image URLs with the appropriate schema validation workflow. Expected result: Referenced images resolve and belong to the entity described on the page. Failure interpretation: Markup points at a missing, blocked, or unrelated asset. Monitoring window: Immediate for markup and HTTP checks; search appearance only after recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial.. Rollback trigger: The change creates invalid markup or associates the wrong image with the entity.

Test yourself: Multimodal Search

Five quick questions on how search worksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank. across text, images, video, and audio. Pick an answer for each, then check.

Resources worth your time

My related writing

• Google AI Overviews: All You Need to Know — how AI answers retrieve from the core indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. (the prerequisite for showing up in multimodal AI results).
• What We Actually Know About Optimizing for LLM Search — the evidence-based view of what does and doesn’t influence AI citationsAn AI citation is the visible source link an AI answer engine shows next to its generated text — the clickable reference that credits the web page it used. A citation's presence is a separate thing from whether the cited page actually supports the statement, and from being retrieved (read behind the scenes) or merely mentioned (named without a link); citation is driven more by brand mentions and being retrievable than by traditional ranking., useful for separating confirmed mechanisms from hype.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and ranking; the pipeline that multimodal retrieval still runs on. (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• MUM: A new AI milestone for understanding information — Google’s own MUM announcement (multimodal, multilingual, 1,000 times more powerful than BERTSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it.).
• How AI powers great search results — Google on how its AI systems, including MUM, fit together.
• Google’s Guide to Optimizing for Generative AI Features — the official line that AI features run on core Search ranking (no separate AI index).
• Google Images SEO best practices — the confirmed image-legibility fundamentals (alt textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search., filenames, structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding.).
• Video SEO best practices — how Google understands video, and the VideoObject / transcript fundamentals.
• Bing Visual Search — Microsoft’s image-as-query surface, a reminder multimodal searchMultimodal search lets you query and get results across more than one modality — text, images, video, and audio together. Instead of typing words, you can point your camera, circle something on screen, or combine an image with a question, and the system understands them jointly. isn’t Google-only.

Videos

• Google Search Central (YouTube) — the How Google Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank. series and Martin Splitt’s explainers on how Google understands images, video, and page content — the machine-understanding side of multimodal searchMultimodal search lets you query and get results across more than one modality — text, images, video, and audio together. Instead of typing words, you can point your camera, circle something on screen, or combine an image with a question, and the system understands them jointly.. Channel

Multimodal Search

Multimodal search lets you query and get results across more than one modality — text, images, video, and audio together. Instead of typing words, you can point your camera, circle something on screen, or combine an image with a question, and the system understands them jointly.

Related: Semantic Search, Embeddings, Retrieval-Augmented Generation (RAG), Grounding

Multimodal Search

Multimodal search is search that works across more than one modality — text, images, video, and audio — either as the input, the output, or both. Where classic search took typed words and returned links, multimodal search lets you point a camera (Google Lens), circle something on your screen (Circle to Search), speak a question, or combine a photo with a text follow-up (“this jacket, but in green”), and the system interprets all of it together rather than one channel at a time.

The technical enabler is shared representation. Modern models encode images, text, audio, and video into the same embeddingEmbeddings are dense numerical vectors — lists of floating-point numbers — that represent the meaning of text in a high-dimensional space. Semantically similar content lands close together, so search and AI systems can match by meaning, not just keywords. space, so a picture of a plant and the words “how to care for this plant” can be compared and matched even though they started as completely different kinds of data. That’s the same meaning-based retrieval behind semantic searchSemantic search is meaning-based retrieval — matching what a user means, not just the words they typed. Search engines detect entities, expand synonyms, infer intent, and rank by conceptual relevance, which is why keyword stuffing lost its power and topical depth gained it., extended past text. Google’s MUM (Multitask Unified Model, 2021) was Google’s public marker for this shift — described as multimodal and able to understand information across text and images — and today’s AI Mode and Gemini-powered features push it further, letting a single visual or spoken query fan outQuery fan-out is the technique where an AI search system breaks a single user question into multiple related sub-queries, runs those searches concurrently, and synthesizes the retrieved results into one answer. Google confirms AI Overviews and AI Mode 'may use a query fan-out technique' issuing multiple related searches across subtopics. into an answer that pulls from many sources.

Two distinctions matter. First, multimodal search is broader than “image search”: image search finds pictures, whereas multimodal search uses a picture (or voice, or video) as part of a query that can return any kind of result. Second, most of what search engines have officially confirmed concerns understanding the query across modalities — how much the content on your page is understood image-by-image or frame-by-frame at web scale is only partly documented, so treat sweeping “optimize your images for multimodal ranking” claims as industry theory, not confirmed mechanism. The durable takeaways are the same ones that already help: descriptive alt textAlt text is the value of the `alt` attribute on an HTML `<img>` element — a short text substitute chosen for the image's purpose and context, not a literal description of what it shows. It makes images accessible to screen-reader users and helps search engines understand images, mainly for image search. and filenames, real transcripts and captions for audio and video, clean structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding., and content that clearly answers the question a visual or spoken query implies.

Related: Semantic Search, Embeddings, Retrieval-Augmented Generation (RAG), Grounding

Changelog