These are the documented Google crawlers:
Googlebot
Googlebot Image
Googlebot Video
Googlebot News
Google StoreBot
Google-InspectionTool
GoogleOther
GoogleOther-Image
GoogleOther-Video
Google-CloudVertexBot
Google-Extended
3. Special-Case Crawlers
These are crawlers that are associated with specific products and are crawled by agreement with users of those products and operate from IP addresses that are distinct from the GoogleBot crawler IP addresses.
List of Special-Case Crawlers:
AdSense
User Agent for Robots.txt: Mediapartners-Google
AdsBot
User Agent for Robots.txt: AdsBot-Google
AdsBot Mobile Web
User Agent for Robots.txt: AdsBot-Google-Mobile
APIs-Google
User Agent for Robots.txt: APIs-Google
Google-Safety
User Agent for Robots.txt: Google-Safety
3. User-Triggered Fetchers
The User-triggered Fetchers page covers bots that are activated by user request, explained like this:
“User-triggered fetchers are initiated by users to perform a fetching function within a Google product. For example, Google Site Verifier acts on a user’s request, or a site hosted on Google Cloud (GCP) has a feature that allows the site’s users to retrieve an external RSS feed. Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. The general technical properties of Google’s crawlers also apply to the user-triggered fetchers.”
The documentation covers the following bots:
Feedfetcher
Google Publisher Center
Google Read Aloud
Google Site Verifier
Takeaway:
Google’s crawler overview page became overly comprehensive and possibly less useful because people don’t always need a comprehensive page, they’re just interested in specific information. The overview page is less specific but also easier to understand. It now serves as an entry point where users can drill down to more specific subtopics related to the three kinds of crawlers.
This change offers insights into how to freshen up a page that might be underperforming because it has become too comprehensive. Breaking out a comprehensive page into standalone pages allows the subtopics to address specific users needs and possibly make them more useful should they rank in the search results.
I would not say that the change reflects anything in Google’s algorithm, it only reflects how Google updated their documentation to make it more useful and set it up for adding even more information.
Read Google’s New Documentation
Overview of Google crawlers and fetchers (user agents)
List of Google’s common crawlers
List of Google’s special-case crawlers
List of Google user-triggered fetchers
Featured Image by Shutterstock/Cast Of Thousands