Agentic Search

TL;DR — Agentic search is AI that does the whole research job for someone, not just answers one question. Instead of a quick reply, it goes off and runs dozens of searches, reads the actual pages, and comes back with a written-up answer — sometimes minutes later. Think ChatGPT giving you a fast answer vs. ChatGPT Deep Research disappearing for ten minutes and returning a report. For you, the practical thing to know: some of these agents browse as a real Chrome browser, so robots.txt can’t stop them.

The three layers of search

“Agentic search” is an evolving product pattern rather than a single standardized ranking architecture. Evidence for this claim Deep-research products can perform multi-step web research and synthesize cited results. Scope: Documented product capabilities, not a universal definition or ranking architecture for agentic search. Confidence: high · Verified: 2026-07-14 OpenAI: Introducing deep research Documented systems may browse, synthesize, and take multi-step actions, but behavior varies by product and release. Evidence for this claim The term agentic search covers an evolving set of vendor implementations whose tools and behavior can differ. Scope: Editorial synthesis from documented product behavior; implementations and releases may vary. Confidence: medium · Verified: 2026-07-14 OpenAI: Introducing deep research

It helps to think of search as having grown three layers, one on top of the next:

1. Traditional search. You type a query, Google hands back a list of links, and you read them and decide. The human does the work.
2. AI search. You ask a question, the AI pulls in some sources and writes you one answer — an AI Overview, a ChatGPT reply, a Perplexity answer. The AI reads, you read its summary.
3. Agentic search. You give the AI a goal — “compare the best project management tools for a 10-person agency and tell me which to pick” — and it goes off on its own. It plans the work, runs many searches, opens and reads real pages, changes its approach based on what it finds, and finally writes you a report (or even takes an action, like filling a form). You might never see the steps in between.

The cleanest way to remember the relationship: all agentic search is AI search, but not all AI search is agentic. Agentic is the subset that goes off and works on its own for a while.

A concrete example

Same product, two modes:

• ChatGPT, regular answer: you ask something, it replies in a few seconds from what it already “knows” plus maybe a quick web look. One pass.
• ChatGPT Deep Research: you ask the same thing, it says it’ll take a while, and it spends minutes searching, reading, and cross-checking dozens of sources before it writes up a structured answer with citations.

That second mode is agentic search. Google’s Gemini Deep Research and Perplexity’s Deep Research work the same way.

Why an SEO should care

Two reasons:

• These agents read your actual pages, deeply. A traditional crawler skims a lot of pages shallowly. An agent might read 5–10 pages very thoroughly and pull specific claims out of them. Being one of those few pages is the new game.
• You can’t always keep them out. The “research crawler” versions (like Gemini’s) announce themselves and follow your robots.txt. But the agentic browser versions — ChatGPT Agent, Google’s Project Mariner — open a real Chrome browser in the cloud and look exactly like a person visiting. There’s no bot name to block, so robots.txt does nothing.

Want the full breakdown — every platform, the user-agent table, and what content signals actually help — switch to the Advanced tab.

TL;DR — Agentic search is the third layer of search (traditional → AI search → agentic): a system that runs a multi-step loop — Plan → Search → Read → Evaluate → re-search → synthesize/act — over minutes, firing 20–160+ queries per session and reading full page content, not cached snapshots. The platforms split into two camps for control purposes. Declared crawlers (Gemini-Deep-Research, Claude-User) respect robots.txt. Agentic browsers (ChatGPT Agent, Project Mariner, Copilot Actions) spin up real Chrome/Edge VMs with no declared bot UA — robots.txt can’t touch them; only server-side controls or the emerging Web Bot Auth RFC can. The payoff: AI-referred traffic converts ~42% better than non-AI, and agentic traffic is the highest-intent slice of that.

Agentic search is a third layer, not a replacement

Current vendor descriptions support multi-step research and tool use, not a universal replacement for retrieval or ranking. Evidence for this claim Deep-research products can perform multi-step web research and synthesize cited results. Scope: Documented product capabilities, not a universal definition or ranking architecture for agentic search. Confidence: high · Verified: 2026-07-14 OpenAI: Introducing deep research Pipeline diagrams here are explanatory models unless explicitly documented by a provider. Evidence for this claim The term agentic search covers an evolving set of vendor implementations whose tools and behavior can differ. Scope: Editorial synthesis from documented product behavior; implementations and releases may vary. Confidence: medium · Verified: 2026-07-14 OpenAI: Introducing deep research

The mental model I keep coming back to:

• Traditional search returns links; the human reads and decides.
• AI search (the GEO/AEO target — AI Overviews, ChatGPT answers, Perplexity) returns one answer synthesized from retrieved context.
• Agentic search breaks a goal into sub-tasks, picks its own tools, reads full pages, iterates on what it learns, and ends in a synthesized report or an action (a purchase, a booking, a form fill).

Semrush put the relationship as well as anyone: “All agentic search is AI search. Not all agentic search is the same thing as AI search.” The short version that sticks — all agentic search is AI search; not all AI search is agentic. None of this replaces traditional SEO. Authority, indexability, and structured data are still prerequisites; agentic optimization stacks completeness, machine-readability, and entity clarity on top.

How agentic systems actually work

The defining difference from RAG or a single retrieval call is that retrieval happens mid-reasoning, on a loop:

Firecrawl describes it as a method where the agent “decides when to search, formulates the query from context, reads actual page content from each result, and evaluates what it found before deciding whether to synthesize or search again.” The practical consequences:

• Real-time page reads, not pre-indexed summaries. Agents fetch and read full live content. Freshness and live availability matter more than for traditional search; a cached snapshot isn’t what they’re working from.
• Iterative query reformulation. If results are stale or off-target, the agent rewrites the query (e.g. tacking “2026” onto a query when it gets deprecated docs) and goes again.
• Tool chaining. A single task can combine web search, code execution, file reads, and form submission.

How many searches per session

This is not one query. Volume by platform:

Google’s SAGE research found AI agents take an average of 4.9 steps per query — searching, comparing, and evaluating across multiple sources before delivering a result.

The major agentic platforms

OpenAI — Deep Research + ChatGPT Agent. Deep Research (o3-deep-research / o4-mini-deep-research) is built to “find, analyze, and synthesize hundreds of sources to create a comprehensive report at the level of a research analyst,” and OpenAI warns the requests “can take a long time.” ChatGPT Agent is the successor to Operator — a cloud-hosted Chromium instance that browses the real web and can run ~10 parallel tasks. Its retrieval bots (OAI-SearchBot, ChatGPT-User) are declared; the agentic browser is not.

Google — Gemini Deep Research + Project Mariner. Gemini Deep Research runs an explicit “Plan → Search → Read → Iterate → Output” loop — Google describes it “continuously refin[ing] its analysis, browsing the web by searching and finding interesting pieces of information then starting a new search based on what it’s learned, repeating this process multiple times before generating a comprehensive report.” It has a declared Gemini-Deep-Research user agent and respects robots.txt. Project Mariner is the browser-automation agent (up to 10 parallel tasks) — it runs a standard Chrome fingerprint from cloud VMs with no declared bot UA.

Perplexity — Deep Research / Sonar. Perplexity “performs dozens of searches, reads hundreds of sources, and reasons through the material to autonomously deliver a comprehensive report” — 20–50 queries in 2–4 minutes. Its declared PerplexityBot says it respects robots.txt. In August 2025, Cloudflare reported observing undeclared crawlers it attributed to Perplexity reaching content despite no-crawl directives; Perplexity disputed Cloudflare’s account. Separately, Perplexity’s docs say Perplexity-User generally ignores robots.txt because it represents a user-requested fetch.

Anthropic — Claude web search + computer use. Claude-User (real-time user-initiated fetches) respects robots.txt. When Claude runs in computer-use mode it drives a standard browser and emits no declared bot UA — the same agentic-browser pattern as Mariner and ChatGPT Agent.

Microsoft — Copilot + Copilot Actions. Copilot’s underlying web retrieval uses Bingbot, which respects robots.txt. Copilot Actions (agentic browsing) uses a standard Edge/Chromium UA with no declared bot signal — indistinguishable from human browsing at the UA layer.

User agents and the robots.txt gap

This is the part that trips people up, so here’s the whole landscape in one table:

The categories that matter: training crawlers and search-index bots respect robots.txt; user-triggered fetchers vary (OpenAI quietly muddied ChatGPT-User’s position in December 2025); and agentic browsers ignore it by design.

The key gotcha: agentic browsers are unblockable via robots.txt

robots.txt is a directive for crawlers. Agentic browsers — ChatGPT Agent, Project Mariner, Copilot Actions, computer-use Claude — spin up real Chrome or Edge instances inside cloud VMs. They present as standard browsers, declare no bot identity, and are treated as browser proxies rather than crawlers. So:

• You can block training (GPTBot, ClaudeBot) and search-index bots (OAI-SearchBot, Claude-SearchBot) in robots.txt.
• You cannot block the agentic browsers there. They simply don’t read it.

The only things that work against the agentic-browser layer are server-side controls (IP-range rules, rate limiting, bot-management at the edge) and the emerging Web Bot Auth RFC — cryptographic agent verification. ChatGPT Agent is already an early signal here: it sends an ed25519-signed Signature-Agent: "https://chatgpt.com" HTTP Message Signature, which is the only reliable server-side identifier for it (in plain analytics it shows up as “direct/(none)” on a direct visit, or “Bing/organic” when it found you via Bing search).

One corollary worth saying out loud: blocking GPTBot does not stop ChatGPT from reading your pages in real time. GPTBot is the training crawler. Deep Research and ChatGPT Agent reach your content through entirely different mechanisms.

What content signals matter for agentic AI

Agents read full pages and evaluate them for completeness, citeability, consistency, and machine-readability. From the research, what actually helps:

• Comprehensive, single-resource pages. Agents go deep but narrow — they may read 5–10 pages thoroughly rather than skim hundreds. A page that fully answers a topic (and the sub-questions it raises) is far more useful than five thin ones. ALM Corp’s framing: “Structure gives agents stable extraction points — a page with clear headings, labeled sections and consistent formatting is easier to quote accurately.”
• Well-cited, authoritative, machine-readable content. Agents cross-reference against third-party sources and knowledge bases, so external citations help you clear their trust threshold. This lines up with what I found for AI Overviews: mentions on heavily-linked pages correlate at ~0.70 (Spearman) with AI visibility — the strongest single signal in that study, and a pattern I’d expect agentic retrieval to lean on too.
• Structured data. JSON-LD isn’t required for citation, but it improves extraction accuracy — GPT-4 went from 16% to 54% correct responses when content carried structured data in one cited test.
• Speed. BrightEdge data shows ChatGPT agents abandon slow-loading sites immediately — they have less patience than Googlebot, which extends Core Web Vitals discipline into the agentic context. Content also needs to be reachable without JavaScript execution.
• Scope and tradeoff clarity. Pages that state who a solution is for, who it’s not for, and the tradeoffs are more useful to an agent building a shortlist than content trying to appeal to everyone.

Why this is worth the effort: conversion quality

The traffic numbers are small but the quality is not. As of March 2026, AI-referred traffic converts about 42% better than non-AI traffic, and AI search traffic converts at roughly 14.2% vs. Google’s 2.8%. Agentic traffic is a subset of AI traffic — and almost certainly the highest-intent slice, because an agent only lands a user on your page after evaluating it against alternatives on the user’s behalf. Being one of the handful of pages an agent reads, and being the one it recommends, is disproportionately valuable.

How to think about it: SEO → GEO → ASO

Treat agentic optimization as a third layer stacked on the first two, not a replacement. The foundations don’t change — authority, indexability, structured data. What’s new on top: comprehensive single-topic resources, ruthless machine-readability, entity clarity, external citations, and the recognition that robots.txt is no longer a complete access-control story. See the Checklists tab for the concrete pass.

AI summary

A condensed take on the Advanced version:

• Agentic search is search’s third layer: traditional (links) → AI search (one answer) → agentic (autonomous multi-step research/action). “All agentic search is AI search. Not all AI search is agentic.”
• It runs a loop, not a query: Plan → Search → Read → Evaluate → re-search → synthesize/act, over minutes. Gemini Deep Research fires ~80–160 queries/session; Perplexity 20–50; Google’s SAGE found ~4.9 steps per query.
• Two control camps. Declared crawlers (Gemini-Deep-Research, Claude-User) respect robots.txt. Agentic browsers (ChatGPT Agent, Project Mariner, Copilot Actions) run real Chrome/Edge VMs with no declared bot UA — robots.txt can’t block them.
• The gotcha: only server-side controls or the emerging Web Bot Auth RFC stop agentic browsers. ChatGPT Agent’s ed25519 Signature-Agent header is the only reliable server-side ID. Blocking GPTBot ≠ stopping ChatGPT from reading you.
• Cloudflare and Perplexity dispute what happened. Cloudflare reported that undeclared crawlers it attributed to Perplexity bypassed no-crawl directives; Perplexity denied the allegation. Treat that episode as a reported dispute, not a settled finding.
• What helps: comprehensive single-topic pages, external citations (~0.70 Spearman with AI visibility), structured data, speed (agents abandon slow sites), scope clarity.
• Why bother: AI-referred traffic converts ~42% better; AI search 14.2% vs. Google’s 2.8%. Agentic is the highest-intent subset.

Official documentation

Primary-source documentation from the platform makers.

OpenAI

• Deep Research guide — the o3-deep-research / o4-mini-deep-research models, background execution, and tool access.
• OpenAI bots & crawlers — full user-agent strings for GPTBot, OAI-SearchBot, and ChatGPT-User.
• Introducing ChatGPT Agent — the agentic browser successor to Operator.

Google

• Gemini Deep Research API docs — query counts, duration limits, and background execution.
• Try Deep Research in Gemini — Google’s plain-language description of the research loop.

Anthropic

• Claude computer-use tool — how Claude drives a browser/desktop in agentic mode.

Perplexity

• Sonar Deep Research — the API model behind Deep Research.
• Perplexity crawlers — PerplexityBot vs. Perplexity-User and their stated robots.txt behavior.

Microsoft

• Bing Webmaster Tools — Bingbot underpins Copilot’s web retrieval; crawl control lives here.

Quotes from the source

On-the-record statements from the platform makers and the researchers documenting their behavior.

Google — the Gemini Deep Research loop

• “Over the course of a few minutes, Gemini continuously refines its analysis, browsing the web by searching and finding interesting pieces of information then starting a new search based on what it’s learned, repeating this process multiple times before generating a comprehensive report.” — Google Blog. Source

OpenAI — what Deep Research is for

• Deep Research models “find, analyze, and synthesize hundreds of sources to create a comprehensive report at the level of a research analyst.” [unverified] — paraphrase of OpenAI’s Deep Research positioning; confirm against the live doc. Source

Perplexity — how Deep Research works

• “Perplexity performs dozens of searches, reads hundreds of sources, and reasons through the material to autonomously deliver a comprehensive report.” — Perplexity Blog. Source

Cloudflare — Perplexity’s stealth crawlers

• “Both their declared and undeclared crawlers were attempting to access the content for scraping contrary to the web crawling norms as outlined in RFC 9309.” — Cloudflare. Source

Firecrawl — defining agentic search

• Agentic search is a method where the agent “decides when to search, formulates the query from context, reads actual page content from each result, and evaluates what it found before deciding whether to synthesize or search again.” — Firecrawl Blog. Source

ALM Corp — what agents reward

• “Structure gives agents stable extraction points — a page with clear headings, labeled sections and consistent formatting is easier to quote accurately.” — ALM Corp. Source

Semrush — the layer distinction

• “All agentic search is AI search. Not all AI search is agentic.” [unverified] — paraphrase of Semrush’s framing; confirm exact wording against the live post. Source

Agentic-search optimization checklist

A pass to make sure agents can find, read, trust, and cite you — and that you can see them when they do:

• Comprehensive single-topic pages. Each important topic has one page that fully answers it (and the sub-questions it raises), rather than several thin pages. Agents read deep, not wide.
• Cited sources and external authority. Claims are attributable; you’re mentioned on heavily-linked, high-authority third-party pages (the strongest AI-visibility signal I’ve measured).
• Structured data in place. JSON-LD (Organization, FAQPage, Product, etc.) so agents extract facts accurately without guessing.
• Scope and tradeoff clarity. Pages state who a solution is for, who it’s not for, and the tradeoffs — easier for an agent building a shortlist.
• Reachable without JavaScript and fast. Core content in raw HTML; pages load quickly (agents abandon slow sites faster than Googlebot does).
• Server-side access controls, not just robots.txt. If you need to gate access, use IP-range rules / edge bot-management — agentic browsers ignore robots.txt.
• robots.txt still set correctly for the declared bots. Decide training (GPTBot, ClaudeBot) vs. retrieval (OAI-SearchBot, Claude-SearchBot) vs. deep-research (Gemini-Deep-Research) per your strategy.
• Monitor agentic crawlers in server logs. Watch for the declared deep-research UAs and the Signature-Agent header (ChatGPT Agent’s only reliable ID); remember direct agent visits can show as “direct/(none)” in analytics.

Agentic AI user-agent reference — cheat sheet

The one-line rule: declared crawlers and search bots respect robots.txt; agentic browsers (real Chrome/Edge VMs) don’t — block those server-side or via Web Bot Auth.

Agentic-search mistakes

Assuming robots.txt controls every agent

Some agentic systems browse through ordinary browser infrastructure rather than a declared crawler identity. Use robots rules for bots that honor them, but do not present the file as an access-control boundary.

Designing only for one-shot answers

Agents may compare, revisit, filter, and complete multi-step tasks. Keep navigation, forms, states, prices, policies, and entity relationships explicit enough to survive a longer workflow.

Hiding critical facts behind fragile interactions

An agent cannot reliably use information that appears only after an inaccessible gesture, canvas state, or undocumented client action. Preserve semantic HTML, stable URLs, clear labels, and server-verifiable outcomes.

The Discover, Understand, Act, Verify framework

1. Discover: Important pages and capabilities have stable links or documented machine-readable entry points.
2. Understand: Entities, constraints, prices, policies, and next steps are stated clearly in semantic content.
3. Act: Forms and workflows have explicit labels, predictable controls, and recoverable error states.
4. Verify: A completed action produces a durable confirmation that a user or agent can inspect before continuing.

Use the framework to review a full task journey. A page can be easy to retrieve yet still fail when an agent tries to act or confirm the result.

Tools for agent-accessible search experiences

• SEO MCP Server — inspect a bounded, read-only example of exposing published SEO content and verification capabilities directly to agents.
• robots.txt Tester — verify declared crawler rules and identify the winning directive, while remembering that ordinary browser-based agents may not present a controllable crawler identity.
• Browser developer tools — inspect semantic names, focus order, network calls, error states, and durable completion signals across the task.
• Server logs — distinguish documented bots, ordinary browsers, and unknown clients without claiming that every request identifies an agent or its purpose.

Test yourself: Agentic search

Resources worth your time

Official platform docs

• OpenAI Deep Research and OpenAI bots.
• Google Gemini Deep Research API and the launch blog post.
• Anthropic Claude computer use.
• Perplexity Sonar Deep Research and Perplexity crawlers.

My related writing

• What We Actually Know About Optimizing for LLM Search — the correlation study behind the ~0.70 Spearman finding for mentions on high-authority pages.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — the shifting crawler landscape.

From others

• Cloudflare: Perplexity is using stealth, undeclared crawlers — Cloudflare’s allegation and technical account; Perplexity denied it.
• Simon Willison: ChatGPT agent’s user-agent — the Signature-Agent / ed25519 detail.
• Firecrawl: What is agentic search — the technical anatomy of the loop.
• Semrush: Agentic search and ALM Corp: The Agentic Web Explained — strategy framing.
• No Hacks: The AI User-Agent Landscape in 2026 and SEJ’s AI crawler user-agent list.
• BrightEdge: Agentic AI Activity Doubles — Adapt Your SEO Strategy Now — data on ChatGPT agent activity doubling between July–August 2025 and agents abandoning slow-loading sites faster than Googlebot.
• Seer Interactive: How Will You Know When OpenAI’s Operator Agent Hits Your Website — practical guide to identifying and measuring agentic browser visits in analytics.
• Cloudflare: From Googlebot to GPTBot — Who’s Crawling Your Site in 2025 — Cloudflare’s crawl-volume data across declared AI bots vs. traditional search crawlers.
• Navoto: Agentic Search Optimization — Guide to AI Search Visibility in 2026 — practical ASO framework covering the SEO → GEO → ASO layering and content signals.

Stats worth citing

• AI-referred traffic converts ~42% better than non-AI traffic (March 2026) — the headline reason agentic visibility is worth the work. Source
• AI search converts at ~14.2% vs. Google’s ~2.8% — agentic traffic is the highest-intent subset of that AI traffic. Source
• Gemini Deep Research: ~80 queries per session (standard), ~160 (Max) — the scale of a single agentic research run. Source
• Perplexity Deep Research: 20–50 queries per session, 2–4 minutes — a lighter but still multi-step run. Source
• ~0.70 Spearman correlation between mentions on heavily-linked pages and AI visibility (Google AI Overviews) — my finding, and the likely shape of agentic trust signals too. Source
• Agentic AI traffic grew 1,300% in the first 8 months of 2025, with ChatGPT agent activity alone doubling between July and August 2025 — the fastest-growing slice of AI-referred traffic. Source

Agentic Search

Agentic search is an AI system that autonomously plans, runs, and synthesizes a multi-step research or browsing task — dozens to hundreds of searches and real page reads — instead of returning one answer to a single query. All agentic search is AI search; not all AI search is agentic.

Related: AI Search, AI Crawlers

Agentic Search

Agentic search is an AI system that autonomously plans, executes, and synthesizes a multi-step research or browsing task on a user’s behalf — without the user directing each individual step. It sits a layer above both traditional search and basic AI search: a traditional engine returns a ranked list of links for a human to read; an AI search answer generates one response from retrieved context; an agentic system breaks a goal into sub-tasks, decides which tools to use (web search, page reads, code execution, form-filling), iterates based on what it learns, and either delivers a synthesized report or takes an action.

The shorthand from Semrush captures it: “All agentic search is AI search. Not all AI search is agentic.” The defining trait is the loop — Plan → Search → Read → Evaluate → (re-search) → Synthesize/Act — running over minutes. Gemini Deep Research runs roughly 80–160 search queries per session; Perplexity Deep Research runs 20–50; OpenAI Deep Research can run for tens of minutes.

The technical gotcha that matters for site owners: the agentic browsers in this category (ChatGPT Agent, Google’s Project Mariner, Microsoft Copilot Actions) spin up real Chrome or Edge instances in cloud VMs and browse as standard browsers with no declared bot user agent. They are treated as browser proxies, not crawlers, so robots.txt does not stop them — only server-side controls or the emerging Web Bot Auth verification do. Declared deep-research crawlers like Gemini-Deep-Research and user-fetchers like Claude-User still respect robots.txt; Perplexity’s compliance has been documented as inconsistent.

Related: AI Search, AI Crawlers