AI agents and the new landscape of AI visibility

Between May 2024 and May 2025, the bots crawling the web reshuffled their entire pecking order. Bytespider, the biggest AI crawler a year earlier, collapsed from 42% of AI-only crawl traffic to 7.2%. OpenAI's GPTBot went the other way, from 5% to 30%. That churn is the clearest signal yet that discovery online now runs through AI agents, autonomous software that reads, reasons, and answers on a user's behalf, instead of through a person typing a query into a search box.

For any store or publisher, the job changes with it. You used to compete for a ranking a human would click. Now you're competing to be the source an AI trusts when it answers for that human. We pulled the crawler numbers (Cloudflare, 2025) to show where the traffic actually moved and what it means for your AI visibility.

Key Takeaways

• Discovery is moving from search rankings toward being a trusted, readable source for AI agents that answer on the user's behalf.
• Among AI-only crawlers, GPTBot jumped to a 30% share and Meta-ExternalAgent entered at 19%, while Bytespider fell from 42% to 7.2% (Cloudflare, 2025).
• Googlebot's raw request volume grew 96%, and it now takes 50% of combined search and AI crawl traffic, feeding both classic search and Google's AI Overviews.
• Real-time agent traffic exploded: ChatGPT-User requests rose 2,825% and PerplexityBot 157,490%.
• Only about 14% of top domains manage AI bot access in robots.txt, so most sites have made no deliberate choice at all.

From PageRank to AgentRank: The evolution of digital prominence

For 20 years the scoreboard was your position on a results page. You tuned keywords, chased backlinks, and cleaned up technical performance, all to win a click from a human. AI agents break that model.

An agent doesn't hand back a list of blue links. It reads, synthesizes, and acts: it answers the question, drafts the document, or completes the purchase, drawing on what it already knows and on live calls to the web for anything current.

So visibility stops being about rank and starts being about whether a machine can find your content, parse it, and trust it enough to use. When an LLM ingests your page and treats it as authoritative, your words become part of the answer it gives. That's a deeper kind of presence than sitting at position three.

The data arms race: Fueling the agentic future

The reshuffle among AI-only crawlers shows how hard these companies are competing for training data. OpenAI's GPTBot climbed from 5% to 30%. Meta-ExternalAgent arrived at 19%. The raw material of a good model is broad, current, varied data, and everyone is paying to collect more of it.

AI Agents time series in May 2025. Source: Cloudflare

Bytespider tells the other side of the story. A year earlier it led every AI crawler at 42%. By May 2025 it sat at 7.2%. Lead in this space and you still have to keep spending, on the model and on the pipeline that feeds it, or your agent drifts out of date and out of favor.

AI Agents time series in May 2024. Source: Cloudflare

More crawling buys better agents. A wider, fresher dataset lets an agent ground its answers in verifiable web data instead of guessing, reason across more kinds of source material, and act on current information rather than stale training. So for the companies running these crawlers, visibility means their bots reach the most useful data before a rival's do.

Google's double play: Dominating search and AI visibility

Google is playing its incumbency well. Googlebot, already the dominant search indexer, grew its raw request volume 96% and now pulls 50% of all combined search and AI crawl traffic. That scale does two jobs at once.

Crawling traffic. Source: Cloudflare

It keeps Google's classic search index comprehensive and current. And it feeds Google's own AI features, including AI Overviews, the AI-written summaries that sit above the results and need broad context about the web to work. A separate crawler, GoogleOther, is tagged for research and development, which tells you Google is already collecting for the next generation of agents, not just today's features.

For anyone publishing, the takeaway is blunt. Google now sits on both sides of the shift, so tuning for its AI features and for its search index is one project, not two.

Read more: Why Google’s AI Overviews Demand a New Digital Strategy by 2026

The agentic shift: Hyper-growth in real-time interaction

Top 20 AI and search web crawlers. Source: Cloudflare

The loudest signal comes from the bots that fetch pages live, while a user waits. ChatGPT-User, which fires when ChatGPT reads the web to answer a prompt, grew 2,825%. PerplexityBot grew 157,490%.

Perplexity sells itself as an answer engine: ask a question, get a synthesized, cited response instead of ten links. Growth on that scale means people are skipping the results page and letting an agent read the web for them.

For you, the message is direct. Your content has to be findable and answerable by these agents. When an agent can locate your page, understand it, and use it to answer a question, you've won a spot inside the answer itself. That rewards clean structure, plain semantic meaning, and content solid enough for a machine to quote.

💡Learn more: The Importance of Visibility in AI Agents

The content control crisis: Reclaiming AI visibility

Most sites haven't reacted at all. Only about 14% of top domains set any AI bot rules in robots.txt. The rest sit in a gray zone, letting their pages get scraped for AI training without a decision either way, and usually without payment.

GPTBot sits at the center of the tension. It's both the most blocked bot and the most explicitly allowed one. Block a major AI crawler and you protect your bandwidth and your IP, but you might also cut yourself out of the sources future agents draw from. Allow it and you keep the channel open. Publishers are split down the middle.

Because robots.txt is only a request, control is moving from polite signals to enforcement. Sites are putting AI crawlers behind Web Application Firewalls (WAFs) that can rate-limit or block them outright, and tools like Cloudflare's AI Audit help owners see the traffic and set terms. Visibility now includes deciding how and when an AI gets to use your content, not just whether it can find it.

For businesses, publishers, and content creators, mastering AI Visibility comes down to 5 moves:

• Write for meaning, not just keywords, so an LLM gets a clear, factual answer it can quote.
• Add Schema Markup and other structured data formats so AI systems can read the relationships in your content, not only the words.
• Keep the site fast and technically clean, because slow pages are expensive to crawl and get skipped as a source.
• Set AI crawler access on purpose, using firewalls where it matters and making a real robots.txt decision instead of a default one.
• Watch the traffic. Crawler mixes, model features, and web standards keep moving, so the choices you make now need revisiting.

Is AI recommending your store? Check free.

FAQ on AI agents and AI visibility

What is AI Visibility? It's how easily AI agents and LLMs can find, read, and trust your content. Classic SEO chases human clicks; AI visibility makes sure the systems that answer directly can actually use what you publish.

Why do AI Agents matter? They're fast becoming the front door to information online. If your content isn't built for them, it gets passed over as more people let AI search, summarize, and act for them.

How can I improve AI Visibility? Publish clear, authoritative, structured content. Use Schema Markup, keep the site fast and technically sound, and manage AI crawler access deliberately.

What does the rise in ChatGPT-User and PerplexityBot traffic mean? It means AI systems are pulling and reading web content in real time, a real shift from human-driven search toward AI-mediated discovery.

Can I block AI crawlers? You can ask through robots.txt, but nobody has to honor it. For firmer control, put crawlers behind a Web Application Firewall (WAF) that can restrict or block them.