Crawl budget is a topic people think about when new pages take a long time to get indexed, even though everything is technically in order. For a small site with a couple of hundred pages, this is almost never a problem. But for catalogs, marketplaces, aggregators, and any sites with automatically generated pages, it is one of the factors that can genuinely limit organic traffic growth.
What is crawl budget
Crawl budget is the number of pages on a site that a search bot is willing to crawl within a given period. The budget is not unlimited: Googlebot and YandexBot have limits on the load they put on a specific server (so frequent requests do not take down the site) and an overall resource limit that is distributed among billions of websites.
The budget consists of two components:
- Crawl rate limit — a technical limit on how many requests per second the bot is willing to send without overloading the server. It depends on the site's response speed: the faster the server responds, the more pages the bot can crawl in the same amount of time.
- Crawl demand — the search engine's “demand” to crawl your particular site: how often its content is updated, how authoritative the site is, and how much demand there is for its pages in search results.
If a site is small and does not update frequently, it is extremely unlikely that crawl budget will become a bottleneck at all. Google explicitly says that sites with up to a few thousand pages and a reasonable update frequency do not need to worry about this. The problem arises at scale — tens or hundreds of thousands of URLs, especially those generated automatically.
Where the budget goes
The most common cause is not that the site is “too big,” but that the budget is being spent on the wrong things.
URL parameters and faceted navigation
An online store with filters for price, color, and size can generate thousands of URL combinations for the same set of products: ?color=red&size=42, ?sort=price&page=2, and so on. Each combination is a separate URL that the bot may crawl, spending part of its budget on it, even though the page's content is almost no different from the others.
Duplicate and near-duplicate pages
A similar situation occurs with tags, date archives, paginated pages, and printable versions of pages — these are all technically different URLs with overlapping content.
Automatically generated pages without moderation
This is a more systemic problem: any user action (a search, check, or filter) automatically creates a new indexable page. We recently discussed a case from our own experience in which the feature “publish a check result on a separate page” turned into tens of thousands of URLs with little demand. Read more in our article about scaled content abuse. In this situation, crawl budget goes toward crawling thousands of pages with no value instead of revisiting genuinely important content more often.
Redirects and broken links
Every step in a redirect chain, or attempt to crawl a page that does not exist, also uses up budget. If your site has many broken internal links, you can check them with bulk URL checking.
How to tell whether budget is actually limiting indexing
Signs that this is worth looking into:
- New important pages get indexed in weeks rather than days.
- The number of pages with the status “Discovered – currently not indexed” is growing in Search Console / Yandex.Webmaster.
- Server log files show that the bot is spending a disproportionate number of requests on low-value sections (filters, pagination, and utility parameters).
If none of these apply, budget optimization is not a top priority. It is better to focus on content quality and technical errors.
How not to waste budget
1. Manage crawling through robots.txt
robots.txt prevents a bot from crawling the specified paths — this is a direct way to avoid spending budget on sections that are known to be unimportant (utility parameters, internal search, and the shopping cart). Do not confuse this with noindex: robots.txt does not remove a page from the index if it already has links pointing to it; it simply prevents the bot from accessing it. You can check the current configuration with the robots.txt analyzer.
2. Use canonicalization rather than blocking when at least one version should be indexed
For duplicates that should still be available to users (for example, a page with sorting applied), it is better to use rel=canonical pointing to the primary version rather than block them entirely. This helps the bot understand which version is primary without crawling every variation equally thoroughly.
3. Set up a clean, up-to-date sitemap.xml
A sitemap is a hint to the bot about what to crawl first. If it contains outdated, deleted, or low-quality URLs, this can throw off priorities. You can check the structure and currency of your sitemap with the sitemap generator and validator.
4. Speed up server responses
The faster the server responds, the more pages the bot can crawl in the same amount of time, and the lower the risk that the budget will hit a technical limit rather than a limit imposed by actual demand. For a full technical review of speed and infrastructure, see the site audit.
5. Do not create pages that provide no value
If a feature automatically creates indexable pages (reports, profiles, search results), define a value threshold from the outset: do not index everything indiscriminately; index only pages that meet a minimum engagement or usefulness criterion. It is cheaper to design this in from the start than to clean up thousands of junk URLs a year later.
Checklist
- Check the share of pages marked “Discovered – currently not indexed” in Search Console / Yandex.Webmaster
- Find sections with faceted navigation and URL parameters — block unnecessary combinations in robots.txt
- Set up canonical tags for duplicate page variations that users still need
- Update sitemap.xml and remove outdated and low-quality URLs from it
- Check the site for broken internal links and redirect chains
- Measure server response speed — the bottleneck may be purely technical
- If the site has automatically generated pages, introduce a value criterion for indexing rather than indexing everything indiscriminately
Conclusion
Crawl budget is not an abstract metric for market giants; it is something quite measurable: the ratio of useful pages to the total number of URLs visible to the bot. For most sites, it is enough to tidy up parameters, duplicates, and automatically generated sections once to stop crawl budget from being a problem. Start with a technical audit — it will show the current state of indexing, robots.txt, and the sitemap, and provide a clear baseline for further work.