The advent of large language models (LLMs) marks the most significant shift in the search experience since the rise of mobile technology.
Google’s SERPs are evolving beyond the traditional 10 blue links, introducing AI Overviews and other enhanced features. The full impact on brands and consumers is still unfolding
This article explores the hypothesis that the integration of LLMs into everyday life will fundamentally change how people search.
With LLMs capable of providing detailed answers, consumers are likely to move away from simple queries (e.g., “car insurance,” “running shoes”) toward more complex, conversational searches (e.g., “How much is car insurance for a 2022 Accord?”).
Hypothesis: The shift in consumer search behavior
Since the introduction of LLM models, consumers have started to shift their search patterns for items containing longer queries.
Methodology: Analyzing Google Ads data
Since search data from various LLMs, including Google’s Gemini and AI Overviews, isn’t available yet, I used search query data from Google Ads.
I pulled every search query from Jan. 1, 2022 to May 31, 2024, which resulted in 41 million search queries and over 83 million impressions.
I broke the data down into three buckets:
Pre-ChatGPT (Jan. 1, 2022–Nov. 30, 2022): No such thing yet as consumer-facing LLMs. Consumers would still be searching “traditionally.”
12 months Post-ChatGPT (Dec. 1, 2022–Dec. 31, 2023): During this data range, consumers are now exposed to LLMs and may have begun altering the way they interact with search.
2024 YTD (Jan. 1, 2024–May 31, 2024): Gemini launched on Dec. 6, 2023. This bucket helps keep the months clean, but now with Google’s product fully launched, this would be the most “advanced” they could be given the date range.
Results: How search term lengths and metrics have changed
After looking at 41 million search terms and their corresponding impression, click and cost data over the past 29 months, what can we conclude?
Search term length has increased since the launch of ChatGPT, but it has been slow. CPCs are up, and CTR is down. LLMs and AI Overviews seem to be driving fewer clicks to traditional ads as consumers searching for longer queries may find what they are looking for through the AI Overviews.
Let’s start by looking at how the data has changed over time by query length. Looking at the data from these three datasets, you can see some small shifts over time.
Most notably, there was a shift between keywords with 5-6 words and keywords with 7-8 words. Search terms with 7-8 words in them have nearly doubled since the launch of ChatGPT and have taken that volume from the 5-6 search term bucket. While the rest have stayed fairly consistent, only moving by <15 in either direction.
What could this mean? I would say that those consumers who had previously searched for slightly more complex terms have started to be even more specific with additional conversational modifiers to their queries.
Overall, we are still only talking about 10% of keywords with more than seven words. The majority still have less than four search terms in the query.
Now, let’s examine some of the throughput metrics and how they have changed over time. Starting with CPC, we can see a big spike in CPC costs over the last six months.
All search term buckets have felt this CPC increase, with the biggest jumps being in the 3-6 keyword length buckets. CPCs are up by 60% after being fairly flat over the last 18 months.