Meta is reportedly developing a search engine index for its AI chatbot to reduce reliance on Google for AI-generated summaries of current events. Meta AI appears to be evolving to the next stage of becoming a fully independent AI search engine.
Meta-ExternalAgent
Meta has been crawling the Internet since at least this past summer from a user agent called, Meta-ExternalAgent. There have been multiple reports in various forums about excessive amounts of crawling with one person on Hacker News reporting having received 50,000 hits by the bot. A post in the WebmasterWorld bot crawling forum notes that although the documentation for Meta-ExternalAgent says it respects robots.txt it wouldn’t have made a difference because the bot never visited the file.
It may be that the bot wasn’t fully ready earlier this year and that it’s poor behavior has settled down.
The purpose of the bot is to summarize search results and according to the results it’s to reduce reliance on Google and Bing for search results.
Is This A Challenge To Google?
It may be possible that this is indeed a the prelude to a challenge to Google (and other search engines) in AI search. The information at this time supports that this is about creating a search index to complement their Meta AI. As reported in The Verge, Meta is crawling sites for search summaries to be used within the Meta AI Chatbot:
“The search engine would reportedly provide AI-generated search summaries of current events within the Meta AI chatbot.”
The Meta AI chatbot looks like a search engine and it’s clear that it’s still using Google’s search index.