A report released by Vercel highlights the growing impact of AI bots in web crawling.
OpenAI’s GPTBot and Anthropic’s Claude generate nearly 1 billion requests monthly across Vercel’s network.
The data indicates that GPTBot made 569 million requests in the past month, while Claude accounted for 370 million.
Additionally, PerplexityBot contributed 24.4 million fetches, and AppleBot added 314 million requests.
Together, these AI crawlers represent approximately 28% of Googlebot’s total volume, which stands at 4.5 billion fetches.
Here’s what this could mean for SEO.
Key Findings On AI Crawlers
The analysis looked at traffic patterns on Vercel’s network and various web architectures. It found some key features of AI crawlers:
Major AI crawlers do not render JavaScript, though they do pull JavaScript files.
AI crawlers are often inefficient, with ChatGPT and Claude spending over 34% of their requests on 404 pages.
The type of content these crawlers focus on varies. ChatGPT prioritizes HTML (57.7%), while Claude focuses more on images (35.17%).
Geographic Distribution
Unlike traditional search engines that operate from multiple regions, AI crawlers currently maintain a concentrated U.S. presence:
ChatGPT operates from Des Moines (Iowa) and Phoenix (Arizona)
Claude operates from Columbus (Ohio)
Web Almanac Correlation
These findings align with data shared in the Web Almanac’s SEO chapter, which also notes the growing presence of AI crawlers.
According to the report, websites now use robots.txt files to set rules for AI bots, telling them what they can or cannot crawl.
GPTBot is the most mentioned bot, appearing on 2.7% of mobile sites studied. The Common Crawl bot, often used to collect training data for language models, is also frequently noted.
Both reports stress that website owners need to adjust to how AI crawlers behave.
3 Ways To Optimize For AI Crawlers
Based on recent data from Vercel and the Web Almanac, here are three ways to optimize for AI crawlers.