How does the rapid growth of bots change the open web?
More than half of the web’s traffic comes from bots:
Imperva’s Bad Bot Report 2024 reveals that almost 50% of total internet traffic wasn’t human in 2024. Trend: declining.
Cloudflare’s Radar shows ~70% human traffic and 30% from bots.
Akami reports that 42% of traffic on the web comes from bots.
So far, bots have collected information that makes apps better for humans. But a new species grows in population: agentic bots.
For two decades, we optimized sites for GoogleBot. Soon, we could focus on AI helpers that act as intermediaries between humans and the open web.
We have already optimized for the BotNet with Schema and product feeds in Google’s Merchant Center. XML sitemaps have become table stakes decades ago.
At the next level, we could have separate websites or APIs for agentic bots with a whole new marketing playing field.
Image Credit: Lyna ™
Agentic Web
As we’re getting close to getting more bot than human traffic on the open web, it’s essential to keep in mind that about 65% of bot traffic is estimated to be malicious.
Good bots include scrapers from search engines, SEO tools, security defense, and, of course, AI crawlers. Both types of bots are growing in population, but only the good ones are really useful.
Comparing the last with the previous six months, AI crawlers grew by 18% while search engines slowed down by 10%. GoogleBot specifically reduced activity by -1.6%.
However, Google’s AI crawler made up for it with an increase of 1.4%. GPT Bot was the most active AI crawler, with 3.8% of all requests – and grew 12%.
Google’s AI crawler grew by 62% and was responsible for 3.7% of all bot requests. Given its current growth rate, Google’s AI crawler should soon be the most active on the web.
Image Credit: Kevin Indig
Today, AI bots have three goals:
Collect training data.
Build a search index to ground LLM answers (RAG).
Collect real-time data for prompts that demand freshness.
But currently, all big AI developers work on agents that browse the web and take action for users:
Claude was first with its “Computer Use” product feature: “Developers can direct Claude to use computers the way people do—by looking at a screen, moving a cursor, clicking buttons, and typing text.”
Google’s Jarvis, “a helpful companion that surfs the web for you,” accidentally launched on the Chrome store momentarily.
OpenAI works on “Operator,” also an agent that takes action for you.
I see three possible outcomes:
Agents grow bot traffic on the open web significantly as they crawl and visit websites.
Agents use APIs to get information.
Agents operate on their platform, meaning Operator uses only data from ChatGPT instead of collecting its own.
I think a mix of all three is most likely, with the result that bot traffic would grow significantly.
If that turns out to be true, companies will increasingly build separate versions of their site for bots geared towards speed and structured data. “Just focus on the user” becomes “just focus on the agents.”
In a way, ChatGPT Search is already an agent that browses and curates the web for humans. But if agents keep their promise, you can expect a lot more.
OpenAI CEO Sam Altman said in his recent Reddit AMA, “I think the thing that will feel like the next giant breakthrough will be agents.”
As bots conquer the web, what are humans doing?
Internet Adoption
Image Credit: Kevin Indig
It’s unlikely that humans stop browsing the web completely.
Even with AI answers in Search and Search features in AI Chatbots, humans still want to verify AI statements, get randomly inspired (serendipity), or seek answers from other humans (Reddit).
But the act of browsing to search will likely dissolve into prompts if bots get good enough.
There is room for growth in human traffic: 70% of the world’s population had internet access in 2023. At the current pace (~7% Y/Y), the whole world would have internet access by 2030.
However, despite growing internet adoption, human traffic has been flat across the last three years (see Cloudflare stats). AI crawlers grow much faster (18%), and agents might speed it up even more.
Image Credit: Kevin Indig
Human requests to Wikipedia, the web’s largest site, have stagnated since 2019.
The reason is that human attention shifts to social platforms, especially for younger generations. As the bots take the open web, humans flee to engagement retreats.
Marketing
Image Credit: Kevin Indig
In the distant future, good bots could become equal rights citizens of the open web nation as more humans spend time on closed social platforms and agentic LLMs increase the already fast growth of bot traffic. If this theory plays out, what does this mean for marketing and specifically SEO?
Imagine booking a trip. Instead of browsing Google or Booking, you tell your agent where you think about going and when.
Based on what it knows about your preferences, your agent curates three options of flights and hotels from your platform of choice.
When you pick a flight, it’s added to your calendar, and the tickets are in your inbox. You don’t need to checkout. The agent does everything for you. You could apply the same scenario to ecommerce or software.
Given that companies like Google already have the capabilities to build this today, it’s worth thinking about what would remain constant, what would change, and what would become more and less important in this vision.