Crawl budget is the number of pages on your site that a search engine is willing and able to crawl within a given period of time.
Search engines don't crawl every page on every site infinitely often — they allocate a rough budget based on factors like how fast your server responds, how important or popular your site is judged to be, and how often your content actually changes. For most small and mid-sized sites, crawl budget isn't a limiting factor at all; it mainly becomes relevant for very large sites with thousands or millions of pages. Wasting crawl budget on low-value pages — thin filtered category pages, infinite calendar archives, duplicate parameter URLs — means a search engine's crawler spends less time discovering and re-crawling the pages that actually matter to your business.
For a large site, an inefficient crawl budget can mean new or updated content takes longer to get discovered and re-indexed, because crawlers are busy revisiting low-value pages instead. For smaller sites, understanding the concept still helps explain why a clean, well-structured site with a sensible sitemap tends to get crawled more efficiently than a messy one.
Illumae's site audit crawls your entire site the same way a search engine does, returning a health score with a prioritised, page-by-page fix list — a useful way to spot the kind of thin or duplicate pages that waste crawl budget on larger sites.
Usually not much. Crawl budget mainly becomes a real constraint for very large sites with thousands of pages. Most small business sites are crawled thoroughly and frequently without any issue.
Google Search Console’s Crawl Stats report shows how often Googlebot is crawling your site and what it’s spending time on, which is the most direct way to check for genuinely wasted crawl activity.
Common culprits are duplicate URLs created by tracking parameters or filters, thin auto-generated pages, and slow server response times that make each individual crawl request more expensive.
No credit card required. See real output for your own business before you pay anything.