Most sites leak crawl budget like a burst pipe. You see it in Search Console — pages discovered but never crawled, orphan URLs sitting invisible, a wall of “Crawled – currently not indexed” that never clears. A technical SEO crawl finds the leaks before Google does. Not a keyword tool. Not a content audit. A raw, structural inspection of every URL your site exposes to search engines, plus a hard look at the paths connecting them. I have run these crawls on sites from 200 pages to 2 million. The same categories of problems surface every time. Here is the process, in order, with the tools that earn their place in your stack.
A technical SEO crawl is an automated scan of your website that mimics how search engine bots discover, fetch, and parse your pages. It maps URLs, status codes, redirects, canonical tags, robots directives, internal links, and rendering behavior — producing a structural health report you can act on.
Why It Matters in 2026
Google’s crawlers are smarter. They are also more selective. The March 2024 core update and the site reputation abuse policies that followed tightened how much crawl attention low-value sections of your site receive. By 2026, crawl budget is a practical concern for any domain carrying more than a few thousand URLs — not just a theoretical one for large sites. Google will not waste resources on pages it has learned to distrust, and that distrust compounds silently. A 2025 study from the Search Engine Land crawl analysis series found that sites with over 30% of URLs returning non-200 status codes saw crawl frequency drop by roughly 40% within two quarters. That is not a penalty. It is neglect, and Google’s response to it looks identical to one.
Rendering compounds the problem. JavaScript-heavy sites now represent the majority of new builds, and Google’s rendering queue is still finite. If your critical content depends on client-side execution, you are asking Googlebot to do extra work for every URL. A crawl that ignores rendered output will miss exactly the problems that matter most.
Step-by-Step
Run these steps in order. Skipping ahead — particularly to step 4 — generates misleading data.
1. Define your crawl scope before you touch a tool. Pull your XML sitemap and your Search Console “Pages” report. Cross-reference them. The URLs that appear in one but not the other are your first investigation targets — they represent either incomplete sitemaps or pages Google knows about that you did not submit. Set your crawler to include the full sitemap plus a representative sample of internal link paths, not just the root domain.
2. Configure your crawler for realistic bot behavior. Set user agent to Googlebot. Enable JavaScript rendering for at least a sample of URLs. Throttle to a crawl rate that will not trigger rate-limiting on your server — if you are on shared hosting, start at two requests per second. Respect robots.txt during the crawl unless you are specifically auditing blocked paths.
3. Crawl and export the raw data. Do not read the summary dashboard first. Export the full URL list with status codes, response times, content types, word counts, and indexability flags. You will filter this file at least six different ways before you are done.
4. Segment by template, not by URL. This is the step most people skip. Group your URLs by pattern — /blog/, /products/, /category/, paginated archives, parameterized URLs. Problems cluster by template. A canonical tag issue on 4,000 product pages is one fix, not 4,000. A crawl that treats every URL as unique will bury you in noise.
5. Audit indexability signals at scale. Check for conflicting directives: noindex tags on pages in your sitemap, canonical tags pointing to redirects, robots.txt blocks on pages receiving internal links. Any of these creates a discovery-to-indexing gap that wastes crawl budget and dilutes signal.
6. Map internal link depth. Every important page should be reachable within three clicks of the homepage. Pages buried deeper receive less crawl attention and pass less authority. Your crawl tool will show you click depth per URL. Sort descending and look at what lives at depth five and beyond.
7. Check redirect chains and loops. A single redirect is fine. A chain of three or more is a crawl inefficiency and a signal leak. Export all 3xx responses and trace each chain to its final destination. Collapse every chain to a single 301.
8. Compare rendered vs. raw HTML. For JavaScript-dependent sites, run two crawls — one with JS rendering disabled, one with it enabled. The delta shows you what content Googlebot sees only after rendering. If critical text, links, or structured data appear only in the rendered version, you have a dependency risk that a rendering queue delay can turn into an indexing failure.
9. Validate structured data and hreflang. Both are crawl-level signals that break silently. Schema errors do not throw visible warnings in Search Console unless you check the enhancement reports. Hreflang return-link errors fragment international signals across regional versions of the same page.
10. Re-crawl after fixes and track deltas. A single crawl is a snapshot. The value is in the comparison. Keep your exports dated and diff them. Crawl frequency in Search Console is your lagging indicator — it should increase within two to four weeks of resolving structural issues.
3 Common Mistakes
Mistake 1: Crawling the live site without throttling. A 50,000-URL crawl at full speed can degrade server response times for real users and, in extreme cases, trigger a temporary crawl slowdown from Google if your server starts returning 5xx errors under load. Fix: throttle to a rate your server can sustain, run crawls during low-traffic windows, and monitor server response times while the crawl runs. If your hosting cannot handle a throttled crawl, that is itself a finding.
Mistake 2: Treating “Crawled – currently not indexed” as a crawl problem. It usually is not. That status typically means Google crawled the page, evaluated it, and decided it was not worth indexing — a quality or duplication signal, not a technical access issue. Fix: before you touch robots.txt or canonicals, compare the non-indexed URLs against your indexed set. If they are thin, duplicative, or templated near-copies of stronger pages, the fix is consolidation or content improvement, not crawl configuration.
Mistake 3: Ignoring crawl budget on faceted or parameterized URLs. E-commerce and directory sites generate thousands of parameter combinations — filters, sort orders, tracking parameters — that create near-duplicate URLs consuming crawl resources. Fix: audit which parameters actually change page content. Block or canonicalize the ones that do not. Do not blanket-block all parameters; some drive legitimate navigation and hold ranking value. Segment by parameter function, not by parameter presence.
Free Tools
Screaming Frog’s free tier caps at 500 URLs, enough to audit a small site or a single template on a large one. It handles JavaScript rendering, custom extraction, and structured data validation without a license.
Google Search Console’s URL Inspection tool gives you the live crawl of a single URL — rendered HTML, discovered links, canonical selection, and indexing verdict. It is the ground truth when your crawler and Google disagree. The “Crawl Stats” report shows crawl requests by response code and file type over 90 days, your best free proxy for crawl budget consumption.
Sitebulb offers a 14-day full-feature trial that includes its crawl budget and internal link analysis modules — worth running once per quarter on complex sites. For log file analysis, GoAccess is free and open-source, and parsing server logs remains the only way to see exactly which URLs Googlebot fetches and how often.
- Cross-reference sitemap URLs against Search Console coverage data before every crawl.
- Segment crawl findings by template pattern, not individual URL — fixes scale that way.
- Run rendered and non-rendered crawls on any JavaScript-dependent site.
- Track crawl frequency in Search Console as your lagging indicator of structural health.
- Collapse redirect chains to single 301s and re-crawl within two weeks to confirm.
Advanced Tip
Correlate crawl frequency with page value. Export your server logs, filter for Googlebot requests over a 30-day window, and join that data against your revenue or conversion data per URL. You will typically find misalignment: high-revenue pages crawled less frequently than low-value blog archives, because the archives generate more internal links and fresher timestamps. Crawl frequency should follow business value. The mechanism for correcting it is internal linking and sitemap priority — not begging Google to crawl more. If your money pages are not getting crawled weekly, the problem is your site architecture, not your crawl settings.
One counter-intuitive point worth sitting with: reducing your total URL count often improves crawl efficiency more than any technical fix. Pruning 20% of low-value pages can increase crawl frequency on the remaining 80% without a single change to robots.txt or canonicals. Most teams resist deletion. The data rarely supports that resistance.
Technical crawls are not a one-time audit. They are a recurring diagnostic. The sites that rank consistently run them on a schedule, diff the results, and fix the template-level issues that compound over time. Run your first crawl this week. Export the data. Segment by template. Fix the highest-impact pattern first. Then crawl again in 30 days and measure the delta. That loop — crawl, segment, fix, re-crawl — is the entire discipline. Everything else is commentary.