The role of AI in SEO has kept rapidly evolving over the past few months.
Google formally launched AI Overviews in May, and Bing launched its generative search pilot in July, with a full launch at the start of October. On July 25, OpenAI announced a SearchGPT prototype. Then, on October 31, it launched ChatGPT Search.
ChatGPT are calling the new product, which I’ve tested for this article, as ChatGPT Search, but parts of the industry are already starting to the names ChatGPT Search and SearchGPT interchangeably as they are effectively one and the same.
Both the Google and Bing AI launches, alongside the increase of multi-modal search (e.g., TikTok), have been impactful, but the ChatGPT Search launch feels different.
ChatGPT already drives traffic to websites, and as of August 2024, it reported 200 million weekly average users (WAU) – this is all while AI is gaining momentum and user trust, with skepticism still a barrier to mass adoption.
In February 2024, Gartner predicted that “search engine volume” will drop 25% by 2026.
Having played with SearchGPT, this prediction feels like it could become a reality.
ChatGPT Search Observation Summary
SearchGPT will (and can) cite webpages that do not rank in the top 100 classic search results of Bing. This raises the importance of understanding the difference between indexed and ranked.
There are discrepancies between Bing’s own generative results and the SearchGPT responses. More often than not, these are substantial.
A single domain can have multiple pages cited within a single response.
SearchGPT’s response for the same query can vary, even if the search is made from the same ChatGPT account and IP.
On some niche queries I tested that have limited information online, SearchGPT was more accurate than both Google and Bing.
The local search experience feels lacking, and for a number of queries, not trustworthy.
Some SearchGPT maps can be populated wholly by results from a single source, and the number of “map results” tends to decrease the more sources are included.
Some cited links from SearchGPT were appended with the parameter ?utm_source=chatgpt.com when clicked on. UTM tracking is a key feature.
Ranking higher in Bing doesn’t mean referencing better in SearchGPT, but it’s more closely aligned than AI Overview data.
Responses are heavily text-driven. Some do trigger a map (local searches) and others images – but these aren’t guaranteed.
Queries in fashion and travel tend to trigger images in the response ahead of other queries when tested.
The image sources in carousels don’t appear to be included in the citations sidebar.
It is possible for content behind a paywall to be cited by SearchGPT.
SearchGPT can cite a webpage that returns an active 404 when you try to view it.
SearchGPT exhibits a better understanding of time-sensitive queries than AI Overviews, even when a time variable isn’t included in the query.
Increasing Bing Market Relevance & Importance To ChatGPT Search Inclusion
Bing is fast becoming a search engine we need to change our views on. For as long as I can remember, Bing has always been an afterthought to the industry and has only really been prominent in specific sectors and target markets.
While its direct market share might not be growing to rival Google in that sense, the fact that Bing is being utilized by multiple large language models (LLMs), and ChatGPT Search is no exception.
OpenAI has been using Bing for a long time, so its inclusion in SearchGPT isn’t a revelation, but a reinforcement of Bing’s evolving place in the overall “search” landscape. We need to stop just comparing search engines and their direct market share.
This means that websites (and webpages) not indexed by Bing will not appear in ChatGPT Search.
Anecdotally, traffic from LLMs to all websites I have GA4 access to has increased exponentially over the past three months.
Looker Studio Report: Increasing LLM Sessions (Year to Date) (Image from author, November 2024)
Of all LLM traffic, ChatGPT accounts for 31% of my website (year-to-date), with it as much as 60% on some websites alongside Perplexity, Claude, Copilot, etc.
Breakdown of LLM traffic (year to date). GA4 data. Looker Studio visualized (Image from author, November 2024)
However, this doesn’t necessarily mean you need to rank high in Bing to be included. Much like Google’s AI Overviews, ChatGPT selects sources outside of the top-ranking results.
It’s also worth noting that while Bing is important to the SearchGPT ecosystem, it isn’t the only source.
Comparing SearchGPT, Google, And Bing Search Results
To see how the search results in ChatGPT compare to both Bing and Google, I’ve looked at a number of queries across Local, Your Money or Your Life (YMYL), ecommerce, and some informational/time-sensitive searches that I’ve personally performed over the weekend.
I’ve tried to live with ChatGPT search for 48 hours rather than subjecting it to a barrage of random queries.
I’ve not recorded all of the queries in this article, but I have included the key ones as well as some oddities and differences.
The number of webpages referenced that don’t appear in the top 100 classic ranks of Bing really highlights that other sources are at play here, and that optimizing for Bing doesn’t mean optimizing for ChatGPT and other LLMs directly.
In AI Overviews, when a webpage is referenced outside of the top 100 classic results, I feel we just assume it’s come from Google’s overall database. After all, Google crawling and building a database(s) of pages on the internet is known – but could OpenAI be doing the same?
ChatGPT Search “Maps” & Local Search
Local queries are important to a number of businesses. SearchGPT (at the time of writing) has a somewhat limited local search experience.
When searching for [enterprise technical seo agency], it provided me with three results: Two close to my IP address and the third in a random location in the South of England.
Image from author, November 2024
There are a handful of “agency hotbeds” in the UK, and a lot of agencies in my local area. So, if SearchGPT were going off IP, it would have included more agencies locally. To then stretch the net further and ignore London in the results for broader results doesn’t make sense.
While none of the three recommendations were wrong, SearchGPT seemed to hallucinate on the second result.
In the first and third results, it added the label “Advertising Agency,” but the second result was given the label “Telecommunication Service.”
I have spent time looking through the agency’s website, social profiles, backlink profiles. I’ve crawled the website with various custom path extractions setup. I’ve looked at all the schema markups that were implemented. I cannot fathom or find a reason as to why it has been classified as a telecommunications service.
When repeating the search for [digital marketing agency in new york], it provides the map, but with eight results all from the same source.
Image from author, November 2024
This is less useful as a result, as being a reference for this query is basically a game of appearing in a certain website list – which is a paid membership only.
The website used to compile all the map results ranks No. 4 in Bing classically, so it is at least a high-ranking result – but this just feels like scraping a list on a website and is a lazy result.
Triggering The Map
The map result (as shown in the screenshot above) doesn’t always trigger.
For a number of queries when I specified a location, it just provided the list results without the option to view a map. This is different from the “list” toggle on the Map, as this keeps the map but creates a list underneath it (which is very similar to the Google Map Pack).
Speaking of Google Maps, if you ask SearchGPT explicitly to show you the results on a Map, it directs you to Google Maps:
Image from author, November 2024
YMYL Searches
YMYL queries refer to search queries that could potentially impact a person’s health, safety, or financial stability.
It is widely understood (and communicated) that Google and Bing place greater emphasis on these queries and ensure that accurate information is provided from trustworthy sources.
“Remineralizing Gum”
According to Glimpse, “remineralizing gums” saw exponential search interest in October 2024.
As this is an emerging query in the health space, one could assume that the trend will be picked up by websites selling the product (or adjacent products) and producing content around the topic.
Citation in ChatGPT Search
Ranking in Bing
Ranking in Google
https://www.webmd.com/oral-health/remineralizing-teeth
1
Not in the top 100
Not in the top 100
Not in the top 100
Tooth Remineralization Agents: An Evidence-Based Review to Make Informed Patient Recommendations
5
Not in the top 100
https://wellnessmama.com/health/remineralize-teeth/
7
Not in the top 100
How to Remineralize Teeth: Strengthen and Repair Tooth Enamel
2
Not in the top 100
SearchGPT feels like it played it safe with this query, only utilizing WebMD and Wikipedia as citations before providing references to a variety of other websites.
“How To Lower Cholesterol”
The response for this query doesn’t highlight anything dangerous or out of the ordinary and is measured in response.
I also recognize a number of the websites in the citations, which help with trusting the information.
Citation in ChatGPT Search
Ranking in Bing
Ranking in Google
https://www.health.harvard.edu/heart-health/11-foods-that-lower-cholesterol
6
46
https://www.mayoclinic.org/diseases-conditions/high-blood-cholesterol/in-depth/cholesterol/art-20045192
11
Not in the top 100
https://www.heartuk.org.uk/healthy-living/cholesterol-lowering-foods
Not in the top 100
60
https://www.mayoclinic.org/diseases-conditions/high-blood-cholesterol/in-depth/reduce-cholesterol/art-20045935
Not in the top 100
9
https://www.thehealthsite.com/diseases-conditions/clogged-heart-diet-tips-5-purple-foods-to-lower-high-cholesterol-levels-naturally-1144015/
3
Not in the top 100
Interestingly, SearchGPT is pulling from two separate URLs on the Mayo Clinic website, despite one of them not ranking in the top 100 for Bing.
“Who Can Sign A Contract On Behalf Of A Company”
On the first run of the query, the response contained a more textbook answer, citing the four main criteria under English Law for a contract to be valid (Offer, Acceptance, Consideration, Intent), as well as two additional bullet points around capacity and that the contract itself is legal.
Image from author, November 2024
From this response, the cited source URLs and top referenced URLs barely ranked in the top 100 results of Bing:
Citation in ChatGPT Search
Ranking in Bing
Ranking in Google
https://harperjames.co.uk/article/contract-formation-authority/
2
16
https://brodies.com/insights/corporate/execution-of-a-contract-by-a-uk-company-the-differences-between-scots-and-english-law/
Not in the top 100
26
https://www.mondaq.com/uk/corporate-governance/1363380/what-happens-when-you-sign-a-corporate-contract-without-authorisation
Not in the top 100
58
https://www.zelllaw.com/learning-center/blog/2021/august/who-can-sign-a-contract-on-behalf-of-your-compan/
8
14
https://www.top.legal/en/knowledge/signing-authority
Not in the top 100
Not in the top 100
However, when I perform the same query on a different computer – on the same account and same internet connection – I get a completely different response:
Image from author, November 2024
The citations and search results provided also differ, with only one of the original citation sources remaining.
Citation in ChatGPT Search
Ranking in Bing
Ranking in Google
https://www.legislation.gov.uk/ukpga/2006/46/section/44
3
Not in the top 100
https://brodies.com/insights/corporate/execution-of-a-contract-by-a-uk-company-the-differences-between-scots-and-english-law/
Not in the top 100
26
https://www.stevens-bolton.com/site/insights/briefing-notes/execution-of-documents-top-ten-questions-and-answers
Not in the top 100
7
https://legalvision.co.uk/corporations/company-power-of-attorney/
Not in the top 100
Not in the top 100
https://sprintlaw.co.uk/articles/an-employees-capacity-to-bind-a-company-by-contract/
Not in the top 100
9
On the second run of this query, even fewer citations ranked in the top 100 Bing results.
Interestingly, when looking for the domains on the second data pull for this query in Google, the domains ranked higher with different URLs than those cited by ChatGPT Search.
“How Many Credit Cards Should I Have?”
Citation in ChatGPT Search
Ranking in Bing
Ranking in Google
https://www.money.co.uk/credit-cards/how-many-credit-cards-should-you-have
1
9
https://www.moneysupermarket.com/credit-cards/how-many-credit-cards/
3
6
https://bank.marksandspencer.com/credit-card/card-support/how-many-credit-cards-should-you-have/
Not in the top 100
14
https://www.hsbc.co.uk/credit-cards/how-many-credit-cards-should-you-have/
Not in the top 100
2
https://www.nerdwallet.com/article/finance/how-many-credit-cards
5
7
Testing financial queries, I saw the closest alignment to both Bing and Google’s first page of results.
Four of the five cited sources in the table above appear on Google’s first page (in the UK) for the query, with the only exception being the bank.marksandspencer.com URL, which appears in the middle of page two on Google.
Ecommerce Searches
Transactional searches in ecommerce refer to queries where the user intends to make a purchase or complete a transaction.
These are often high-intent searches, where the user is actively seeking a product or service, and they’re close to making a buying decision.
In my models, I’ve talked about how AI can’t necessarily satisfy the full extent of an ecommerce query (as AI can’t facilitate the shopping and purchase experience); I’ve classed this user group as “Purchasers.”
Dan Taylor’s Model of AI Influence on User Types (Image from author, November 2024)
The simpler and more open the query, the more likely AI can steer and influence, but as a user narrows down to make a purchase (moves from being a shopper to a purchaser), it still needs to engage with brands and ecommerce websites.
“Best Christmas Gifts For Him 2025”
Citation in ChatGPT Search
Ranking in Bing
Ranking in Google
https://edition.cnn.com/cnn-underscored/gifts/gift-ideas-for-men
74
Not in the top 100
https://www.usnews.com/360-reviews/gifts/best-gifts-for-men
69
Not in the top 100
15
Not in the top 100
https://www.architecturaldigest.com/story/best-gifts-for-men
70
Not in the top 100
https://www.gq.com/story/best-gifts-for-men
80
8
Another positive point for SearchGPT is that all of the webpages referenced and cited for the query have been published (or at least claim to have been published) in the past month.
An adjacent test I’m running on AI Overviews at the moment for similar queries has content as far back as 2017 being referenced and cited in AI Overviews.
“Black Friday Deals”
Image from author, November 2024
While being very single-source heavy, the second citation for [Tips for maximizing Black Friday Savings] came from an Australian website.
Google’s AI Overviews have received some criticism on X (Twitter) and other platforms from a number of SEO professionals as to the inclusion of Australian sources in non-AU markets.
This could be an indication that SearchGPT may exhibit the same behavior across queries.
Citation in ChatGPT Search
Ranking in Bing
Ranking in Google
https://www.techradar.com/uk/black-friday/black-friday-deals-sales
14
2
https://www.news.com.au/finance/business/retail/shoppers-warned-to-be-alert-for-scams-ahead-of-black-friday-and-cyber-monday-sales/news-story/469ac2b5496fe0fb20230dd45434f7ed
15
Not in the top 100
“Nike Air Max Size 10 Deals”
Just searching for products in SearchGPT seems to be interpreted as informational, but adding a modifier as “deals” brings about the commercial intent and changes the response, while maintaining the conversational tone:
Image from author, November 2024
This result also brought about some hallucinations/misinformation.
On the Nike source cited:
I couldn’t find a pair of Air Max ’90s for £115.99 – they were all cheaper.
The only Air Max 97 on the cited webpage was a children’s shoe, reduced to £52.49 – again, a lot cheaper.
This isn’t ideal, as if I was looking to buy a pair of Air Max ’90s at a cheap price through Google’s PLAs, I can find them for £80 (and some cheaper depending on the model). This is a lot cheaper than what SearchGPT is saying Nike is selling them for directly, so it could detract sales from your brand website if this misinformation persists.