Tucked away in the Settings section of Google Search Console is a report few SEO professionals discuss, but I like to monitor.
These reports are known as Crawl Stats.
Here, you’ll find an interesting set of metrics on Googlebot crawl activity. These metrics are especially useful for websites with thousands or millions of pages.
Googlebot ‘Crawl stats’
Long ago, Google Search Console had easy-to-find metrics on Googlebot crawl activity. Then, it seemingly disappeared.
In reality, it was tucked away in the Settings section.
How to access the crawl stats reports:
Click on Settings at the bottom of the left navigation.
Go to the Crawl stats section.
Click Open report.
About the crawl stats data
As Googlebot crawls your site, Google tracks and reports on various aspects of Googlebot’s activity and reports on it in Google crawl stats.
This is where you’ll find high-level statistics about Google’s crawling history on your website.
Google says this data is for advanced users
The Googlebot Crawl Stats data is not for the technical SEO rookies.
Google specifically says this data is aimed at “advanced users” with thousands of pages on their site, which may be why it’s located in such an unusual location, unseen by many in the SEO community.
One reason Google may perceive this as an advanced report is that so many things can influence these metrics, including network issues and cloud delivery services such as Akamai.
Who will find crawl stats most useful?
I find the Crawl Stats reports less of an “advanced” set of reports but something that’s more useful to enterprise SEOs without crawler monitoring tools such as Lumar and Botify.
When doing SEO on an enterprise website with thousands or millions of pages, crawler optimization is a vital task, and crawler activity metrics provide important insight for defining action items.
Smaller sites likely do not need to worry too much about crawler activity because there is probably enough crawl budget allocated to your site to crawl at an appropriate pace.
On the other hand, enterprise sites tend to have far more pages that need to be crawled, discovered, and/or refreshed than Google crawls their site each day.
For this reason, they must optimize for crawler activity, which is a tool to help guide next steps.
What to look for in this data
After years of reviewing this data across many sites, I have one primary rule:
Do not spend a lot of time here unless you see fluctuations and correlations.
Often these reports are interesting but not actionable.
Example fluctuations that I tend to investigate:
HTML requests decreased (or spiked) at the same time Bytes of JavaScript downloaded increased (or spiked).
Average response time increased (or spiked) at the same time the number of HTML requests went down (or sudden fall).
Total download size increased (or spiked), but the number of HTML requests did not change.
The percent of requests for discovery (to discover new URLs) increases and the percent of requests for refresh goes down; however, you did not launch new URLs on the site.
When to look at this crawl stats
Crawl stats are good to peruse (and log) at least once a month.
They are especially good to monitor after major releases, such as a platform migration or major redesign. This will help you quickly understand how Google is responding to your newly launched changes.
Remember: If you have a bot monitoring tool such as Lumar or Botify, then this data isn’t as useful as you’ll find in the bot monitoring data provided by these tools.
Caveats about the crawl stats data
Many things can influence Google’s crawl stats metrics beyond a normal dev release.
This means the SEO team, product manager(s) and developer(s) must think outside the box when evaluating the fluctuations.
You must consider what could have caused an increase or decrease in Googlebot crawl activity, not only in your release but also within the network and tech stack.
Changes to something such as Akamai could potentially impact this data.
Log the crawl stats data in a spreadsheet
This is data I like to archive because Google reports such a small window of time.
A great example of this is a challenge I’m facing now with a client. What is reported in GSC right now looks like things are improving: