It’s not useful content on page /events-calendar/december-2086. There probably haven’t been any events organized yet.
All of the resources wasted on those empty calendar pages could have been utilized by the bots on new products just uploaded to the site.
Accessibility
Search bots may reduce the frequency of crawling a URL if it returns a server response code other than 200.
For example, a 4XX code indicates that the page cannot or should not be found, leading to less frequent crawling of that page.
Similarly, if multiple URLs return codes like 429 or 500, bots may reduce the crawling of those pages and eventually drop them from the index.
Redirects can also impact crawling, albeit to a smaller extent. However, excessive use, such as long chains of redirects, can have a cumulative effect over time.
Get the newsletter search marketers rely on.
How to identify crawl budget problems
It’s impossible to determine if your site is suffering from crawl budget issues by looking at it alone.
See what the search engines are reporting
The first step to identifying if search bots are having issues crawling your site is to use their webmaster tools.
For example, look at the “Crawl stats” report in Google Search Console.
This will help you identify if a problem on your site may have caused Googlebot to increase or decrease its crawling.
Also, have a look at the “Page indexing” report. Here, you will see the ratio between your site’s indexed and unindexed pages.
When looking through the reasons for not indexing pages, you may also see crawl issues reported, such as “Discovered – currently not indexed.”
This can be your first indication that pages on your site do not meet Google’s crawling criteria.
Dig deeper: Decoding Googlebot crawl stats data in Google Search Console
Log files
Another way to tell if the search bots are struggling to crawl your pages as much as they would like to is to analyze your log files.
Log files report any human users or bots that have “hit” your website.
By reviewing your site’s log files, you can understand which pages have not been crawled by the search bots for a while.
If these are pages that are new or updated regularly, this can indicate that there may be a crawl budget problem.
Dig deeper. Crawl efficacy: How to level up crawl optimization
How to fix crawl budget problems
Before trying to fix a crawl budget issue, ensure you have one.
Some of the fixes I’m about to suggest are good practices for helping search bots focus on the pages you want them to crawl.
Others are more serious and could have a negative impact on your crawling if not applied carefully.
Another word of warning
Carefully consider whether you’re addressing a crawling or indexing issue before making changes.
I’ve seen many cases where pages are already in the index, and someone wants them removed, so they block crawling of those pages.
This approach won’t remove the pages from the index – at least not quickly.
Worse, they sometimes double down by adding a noindex meta tag to the pages they’ve already blocked in the robots.txt file.
The problem?
If crawling is blocked, search bots can’t access the page to see the noindex tag, rendering the effort ineffective.
To avoid such issues, don’t mix crawling and indexing solutions.
Determine whether your primary concern is with crawling or indexing, and address that issue directly.
Fixing crawl budget issues through the robots.txt
The robots.txt is a very valid way of helping the search bots determine which pages you do not want them crawling.
The “disallow” command essentially prevents good bots from crawling any URLs that match the disallow command.
Bad bots can and do ignore the disallow command, so if you find your site is getting overwhelmed by bots of another nature, such as competitors scraping it, they may need to be blocked in another way.
Check if your robots.txt file is blocking URLs that you want search bots to crawl. I’ve used the robots.txt tester from Dentsu to help with this.
Improving the quality and load speed of pages
If search bots struggle to navigate your site, speeding up page loading can help.
Load speed is important for crawling, both the time it takes for the server to respond to a search bot’s request and the time it takes to render a page.
Test the templates used on URLs that aren’t being crawled regularly and see if they are slow-loading.
Another reason you may not see pages being crawled, even for the first time, is because of quality.
Audit the pages not being crawled and those that perhaps share the same sub-folder but have been crawled.
Make sure that the content on those pages isn’t too thin, duplicated elsewhere on the site or spammy.
Control crawling through robots.txt
You can stop search bots from crawling single pages and entire folders through the robots.txt.
Using the “disallow” command can help you decide which parts of your website you want bots to visit.
For example, you may not want the search bots wasting crawl budget on your filtered category page results.
You could disallow the bots from crawling any page with the sorting or filtering parameters in the URL, like “?sort=” or “?content=.”
Consider nofollow links on internal links
Another way to prevent bots from crawling certain pages is to add the “nofollow” attribute to the link tag.
With the events calendar example earlier, each “View next month’s events” link could have the “nofollow” attribute. That way, human visitors could still click the link, but bots would not be able to follow it.
Remember to add the “nofollow” attribute to the links wherever they appear on your site.
If you don’t do this or someone adds a link to a deeper page in the events calendar system from their own site, the bots could still crawl that page.
Navigating crawl budget for SEO success in 2025
Most sites won’t need to worry about their crawl budget or whether bots can access all the pages within the allocated time and resources.
However, that doesn’t mean they should ignore how bots are crawling the site.
Even if you’re not running out of crawl budget, there may still be issues preventing search bots from crawling certain pages, or you might be allowing them to crawl pages you don’t want them to.
It’s important to monitor the crawling of your site as part of its overall technical health.
This way, if any issues arise that could hinder bots from crawling your content, you’ll be aware and can address them promptly.
Dig deeper: Top 6 technical SEO action items for 2025
Contributing authors are invited to create content for Search Engine Land and are chosen for their expertise and contribution to the search community. Our contributors work under the oversight of the editorial staff and contributions are checked for quality and relevance to our readers. The opinions they express are their own.