The Always‑Updating Guide to AI Crawlers & User Agents (June 2026)

AI crawlers are quietly reshaping how websites and products show up in search, chatbots, and AI shopping experiences. If you run a store, a content site, or any kind of online business, these bots are already visiting you, even if you've never heard of them.

At Product Registry, we track these crawlers so merchants can see who is hitting their product data, why, and how often. This post is the home for a constantly updated list of AI crawlers and their user‑agent strings.

What are AI crawlers, and why do they exist?

A crawler (or "bot") is a program that visits web pages automatically. It sends a small piece of text called a user‑agent string with every request. That string is how you see "who" is visiting in your logs.

Traditional search engines (like Google or Bing) have done this for years to build search indexes. Now a new class of crawlers does something extra:

• Train AI models - Bots like GPTBot (OpenAI) and ClaudeBot (Anthropic) copy public pages so LLMs can learn from them.
• Power AI search & chat answers - Bots like OAI‑SearchBot (OpenAI), Claude‑SearchBot (Anthropic) and PerplexityBot index content so AI answers can quote or link to your pages.
• Fetch pages on demand for users - Agents like ChatGPT‑User and Claude‑User only load your page when a human asks the AI a question that needs your content.

On top of that, you still have generic crawlers (Googlebot, Bingbot, Applebot) that index the web and then reuse that data inside AI features like Gemini, Bing Copilot, Apple Intelligence, and more.

Most reputable AI crawlers say they respect robots.txt, the simple text file at yourdomain.com/robots.txt that tells bots what they may fetch. But not every crawler is well‑behaved, and some are now accused of stealth crawling (more on that below).

The different types of AI crawlers

Think about AI crawlers in five buckets. This makes it easier to decide which ones you want to allow, limit, or block.

1. AI‑specific crawlers for background tasks (indexing & training)

These bots run in the background, hitting millions of pages a day, and collecting data to train foundation models. Common examples include:

• GPTBot (OpenAI) for training models like ChatGPT.
• ClaudeBot (Anthropic) for training Claude.
• Meta‑ExternalAgent (Meta) for training and indexing content that powers Meta AI across Facebook, Instagram, and WhatsApp.
• DeepSeekBot (DeepSeek) and PanguBot (Huawei) for training their respective foundation models.

One interesting example is the Common Crawl bot, CCBot. Although not branded as "AI", the data it crawls for creating a public dataset of the entire internet, is widely used in AI training pipelines.

These bots usually don't care about "real‑time" freshness. They care about coverage: getting a copy of as much of the public web as possible.

2. AI‑specific crawlers triggered by user actions

These are "on‑demand" bots. They only hit your site when a human uses an AI product:

• ChatGPT‑User - When someone asks ChatGPT something that needs your site, this agent fetches it on their behalf.
• Claude‑User - Same idea for Anthropic's Claude.
• Perplexity (assistant UA) - Used when Perplexity's AI assistant browses specific pages in response to a query.
• Meta‑ExternalFetcher - Fetches a specific page when a Meta AI user (on WhatsApp, Instagram, or Facebook) shares or asks about a URL.
• DuckAssistBot - Activated by DuckDuckGo's DuckAssist feature to grab content from trusted sources.

You can think of these as "AI browser tabs" opened by the user, not background scrapers. They still respect robots.txt (for the major vendors), but blocking them can mean your content won't be referenced when users ask questions.

3. Generic crawlers that also feed AI

These crawlers aren't "AI‑only," but their data increasingly ends up in AI products:

• Googlebot - Primary crawler for Google Search; its index powers Gemini / AI Overviews.
• Bingbot - Primary crawler for Bing Search and Microsoft Copilot.
• Applebot - Apple's crawler for Spotlight, Siri suggestions, and now part of Apple Intelligence.

Most sites keep these allowed, because blocking them means disappearing from traditional search as well.

4. Stealth crawlers (undeclared or misleading)

Here's where things get messy.

Some traffic doesn't clearly identify itself as a bot, or it switches its user‑agent when blocked. Cloudflare and others have published research accusing Perplexity's systems of:

• Crawling blocked sites using generic Chrome browser user‑agents
• Rotating IP addresses and network identifiers
• Ignoring robots.txt and even Web Application Firewall rules

Perplexity disputes parts of those reports, but the takeaway for site owners is simple: some AI traffic may not self‑identify at all. You'll see it as "normal browsers" in your logs.

Bytespider has also been reported as very aggressive, sometimes overwhelming smaller sites, even when they try common blocking methods.

Two newer names belong in this bucket. xAI's Grok publishes no crawler documentation or robots.txt policy, and reports describe its retrieval traffic using spoofed Chrome/Safari user‑agents and rotating residential IPs, so it often looks like a normal browser in your logs. DeepSeek similarly publishes little about its crawler and is frequently reported to fetch content without a clear, blockable user‑agent.

For this class of traffic, robots.txt alone is often not enough; people turn to firewalls, rate limiting, and AI‑specific bot filters.

5. Agentic AI browsers (task‑completing agents)

A newer class of bot doesn't just read pages, it acts on them, clicking buttons, filling forms, and completing multi‑step tasks on a user's behalf:

• Operator / ChatGPT Agent (OpenAI) and GoogleAgent‑Mariner (Google) drive a real browser to complete tasks like research, booking, or purchasing.
• NovaAct (Amazon) and MistralAI‑User (Mistral) offer similar agent‑driven browsing tied to a user request.

Because these agents are triggered by a human in real time, vendors often treat them like user‑initiated fetches, so robots.txt may not fully apply. For site owners, the bigger questions here are usually about checkout flows, rate limits, and whether automated agents should be able to take actions on your site at all.

AI crawler & user‑agent cheat sheet (June 2026)

Below is a starter table of the main AI‑linked crawlers we see most often in logs across the web and in Product Registry data. We'll keep this updated over time.

Important:

• "Respects robots.txt?" is based on official docs + public reporting as of June 2026.
• "Yes" = vendor explicitly documents robots.txt support and no strong evidence to the contrary.
• "Mixed/Contested" = reports of bad behavior or stealth crawling exist.
• Always verify against your own logs and risk tolerance.

*This is not legal advice; it's a snapshot of current documentation and public reports. Always test your own robots.txt and firewall rules.

How this ties back to Product Registry

One of the core jobs of Product Registry is to detect and log AI crawler activity including GPTBot, PerplexityBot, Anthropic bots, Google AI crawlers, and others, so merchants can see which products are being fetched, how often, and by which bots.

Because we publish clean JSON‑LD endpoints and llms.txt/sitemaps, "good" AI crawlers have a clear path to your structured product data, while you stay in control through robots.txt and other controls.

Quick robots.txt example

Here's a simple example of how some sites are starting to treat AI crawlers differently:

# Allow normal search engines
User-agent: Googlebot
Allow: /

User-agent: bingbot
Allow: /

# Allow AI search visibility, but block training
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot
Allow: /
User-agent: Applebot-Extended
Disallow: /

# Block aggressive / unwanted crawlers
User-agent: Bytespider
Disallow: /

User-agent: PerplexityBot
Disallow: /

This is just an example; your ideal setup depends on your business model, your tolerance for scraping, and whether AI visibility is a growth channel or a cost center for you.

What we'll keep updating

Over time we'll keep this page in sync with:

• New AI crawlers and user‑agent strings (especially from smaller AI search engines)
• Changes in behavior (for example, if a bot starts or stops respecting robots.txt)
• Real‑world patterns we see in Product Registry's crawler analytics