Google’s Search Advocate, John Mueller, shared insights on diagnosing widespread crawling issues.
This guidance was shared in response to a disruption reported by Adrian Schmidt on LinkedIn. Google’s crawler stopped accessing several of his domains at the same time.
Despite the interruption, Schmidt noted that live tests via Search Console continued to function without error messages.
Investigations indicated no increase in 5xx errors or issues with robots.txt requests.
What could the problem be?
Mueller’s Response
Addressing the situation, Mueller pointed to shared infrastructure as the likely cause:
“If it shared across a bunch of domains and focuses on something like crawling, it’s probably an issue with a shared piece of infrastructure. If it’s already recovering, at least it’s not urgent anymore and you have a bit of time to poke at recent changes / infrastructure logs.”
Infrastructure Investigation
All affected sites used Cloudflare as their CDN, which raised some eyebrows.
When asked about debugging, Mueller recommended checking Search Console data to determine whether DNS or failed requests were causing the problem.
Mueller stated:
“The crawl stats in Search Console will also show a bit more, perhaps help decide between say DNS vs requests failing.”