The Screaming Frog window froze at 47,312 URLs. A client’s crawl had just stalled because someone set concurrency to 10 and pointed it at a site with 2 million pages and no crawl-delay rules. Four hours gone. That single misconfiguration taught me more about technical SEO crawls than any tutorial. A crawl is not a magic audit button. It’s a diagnostic instrument, and like any instrument it produces garbage if you don’t know what you’re measuring. Over the past decade I’ve run crawls on 40-page local businesses and on enterprise platforms whose faceted navigation could swallow a mid-sized crawler whole. Here’s how to do it without wasting your afternoon.
A technical SEO crawl is an automated process that fetches every accessible URL on a website, simulating how a search engine bot discovers and processes pages. It records status codes, response times, canonicals, meta directives, internal links, and rendering behavior. Then it surfaces the technical issues that block indexing or dilute ranking signals.
Why It Matters in 2026
For large sites, crawl budget stopped being theoretical a while ago. The March 2025 core update tightened how Google evaluates page experience signals, and AI-driven SERP features keep rolling out. The crawlers feeding those systems need clean, fast, unambiguous signals. If your crawl reveals a thousand orphaned pages or a chain of 301s three hops deep, you’re handing Google a map with dead ends drawn on it.
Here’s what changed. Rendering budgets are tighter. Googlebot’s willingness to execute JavaScript has always been conditional, and sites that lean on client-side rendering for critical content get indexed less reliably than they did two years ago. A crawl that only reads raw HTML misses half the picture. You need to compare raw and rendered output, which most people skip.
My counter-intuitive take: the crawl itself is rarely where the value lives. The value is in what you do with the crawl diff. Running the same crawl monthly and comparing exports catches regressions that a one-off audit never will. Sites don’t break all at once. They rot one deploy at a time.
Step-by-Step
1. Define scope and seed URLs before you touch a tool. Decide whether you’re crawling the whole domain, a subfolder, or a staging environment. Pull your XML sitemaps and your most important internal navigation paths. A crawl seeded only from the homepage will miss orphaned sections buried deep in the architecture. Seed from sitemaps plus a handful of high-traffic URLs you know exist.
2. Configure robots.txt and user-agent behavior deliberately. Test your robots.txt in the tool’s built-in tester. Decide whether you’re crawling as Googlebot desktop, Googlebot smartphone, or a generic user agent. In 2026, mobile-first indexing means smartphone Googlebot is the one that matters. Crawl as the wrong agent and you get a false picture of what Google actually sees.
3. Set crawl speed and concurrency to match the host. On shared hosting, drop concurrency to 1–2 and add a delay. On a dedicated server with headroom, 5–10 concurrent connections is fine. Watch server response times during the crawl. If they climb above 500ms consistently, you’re the bottleneck, and your crawl data is now polluted by your own pressure.
4. Enable JavaScript rendering for at least a sample. Run a raw HTML crawl first, then a rendered crawl on a subset. Export both and diff the title tags, canonical tags, and internal links. Any link that exists only after rendering is a link Googlebot may or may not follow, depending on rendering budget. That gap is where modern indexation problems hide.
5. Audit status codes and redirect chains. Filter for 4xx and 5xx first. Then pull every 3xx and trace the chain length. Anything beyond one hop wastes crawl budget and dilutes signal. Map where each redirect terminates and confirm the final URL returns 200 and is canonical.
6. Check indexability directives at scale. Export every page’s meta robots, X-Robots-Tag headers, and canonical tags into one sheet. Look for conflicts: a page that’s canonicalized away but also in the sitemap, or a noindex tag on a page receiving internal links. These contradictions confuse crawlers and waste budget.
7. Analyze internal link distribution. Sort pages by internal inlinks. Your money pages should be near the top. Pages with zero inlinks that aren’t in the sitemap are orphans. Pages with hundreds of inlinks from boilerplate navigation may be over-linked. Look at anchor text diversity too. A site where every internal link says “click here” is telling Google nothing.
8. Measure page weight and Core Web Vitals proxies. Most crawlers capture response size, load time, and resource counts. You won’t get real field data, but you’ll spot the outliers: the 8MB images, the pages pulling 47 scripts, the ones timing out. Flag them for a Lighthouse pass.
9. Export, segment, and prioritize. A raw export is noise. Segment by template, by section, by status code. Rank issues by traffic exposure. A noindex on a page with 40,000 monthly sessions outranks a broken link in the footer. Build a fix list ordered by estimated impact, not by how easy the fix is.
10. Schedule the next crawl before you close the laptop. Set a recurring crawl and store each export with a date stamp. The diff between this month and last month is your regression alarm. Deploy cycles break things quietly. Recurring crawls catch them.
3 Common Mistakes
Mistake 1: Crawling production at full speed and calling it a day. Blasting a live site with 10 concurrent connections on shared hosting can trigger rate limiting, WAF blocks, or genuine downtime. Your crawl then returns 429s and timeouts that look like site errors but are artifacts of your own aggression. Fix: throttle to match the host, monitor response times mid-crawl, and re-crawl any section that returned errors before you report them as issues.
Mistake 2: Ignoring rendered vs. raw HTML entirely. If your site uses React, Vue, or any client-side framework for critical content, a raw crawl tells you almost nothing about what Google indexes. I’ve seen sites where the raw HTML had zero internal links in the main content and the rendered version had hundreds. Fix: always run a rendered crawl on a representative sample and diff the outputs. Treat any content that appears only after JavaScript as a risk to quantify, not a given.
Mistake 3: Treating the crawl export as the deliverable. A 12,000-row spreadsheet is not an audit. It’s raw material. Stakeholders don’t act on “1,847 pages return 404.” They act on “the product pagination template is generating 1,847 dead URLs, and fixing it recovers crawl budget for 12,000 product pages.” Fix: translate every issue into traffic exposure, effort, and owner. Rank by impact. Kill the rest.
Free Tools
Screaming Frog SEO Spider (free tier). The standard. The free version caps at 500 URLs, which is plenty for small sites and for sampling large ones. It handles rendering, custom extraction, and structured data validation. If you learn one crawler, learn this one.
Google Search Console. Not a crawler, but the ground truth for what Google actually indexed. Compare your crawl’s indexable page count against GSC’s indexed count. A big gap means something between your crawl and Google’s index is broken. The URL Inspection tool also shows you the rendered HTML Googlebot saw for any single URL.
Sitebulb (free trial, then paid) and Seolyzer. Worth mentioning because both surface issues with plain-language explanations rather than raw columns. Good for teams where not everyone reads status codes fluently.
curl and the command line. Free, instant, and underused. A quick curl -I against a URL tells you the status code, redirect target, and headers in under a second. For spot-checking a handful of URLs, it beats spinning up a full crawl.
XML Sitemap validators and the robots.txt tester in GSC. Both free, both catch the configuration errors that break crawls before they start.
- Always throttle crawl speed to the host’s capacity and watch response times mid-crawl.
- Run both a raw HTML and a rendered crawl, then diff the results for content and link gaps.
- Export every indexability signal (meta robots, X-Robots-Tag, canonical, sitemap status) into one comparison sheet.
- Rank fixes by traffic exposure and effort, not by how quickly they can be closed.
- Schedule recurring crawls and diff exports month over month to catch regressions.
Advanced Tip
Build a crawl-to-log-file comparison. Your server access logs record every request Googlebot actually made, with timestamps, status codes, and response sizes. Your crawl records what’s theoretically reachable. The gap between them is the most valuable diagnostic you have. If Googlebot is hammering URLs your crawl never found, those URLs exist and matter. If Googlebot never touches pages your crawl says are important, you have a discovery problem, not a content problem. Export the log, filter for Googlebot user agents, and cross-reference against your crawl export. Most SEOs never do this. The ones who do find indexation issues weeks before Search Console reports them.
One more thing worth doing: crawl your staging environment with authentication headers so you catch issues before they ship. A crawl that runs post-deploy is a post-mortem. A crawl that runs pre-deploy is prevention.
Conclusion
A technical SEO crawl is only as good as the questions you bring to it. The tool fetches; you interpret. Configure it carefully, render it properly, and treat the export as the start of the analysis rather than the end. Do you know what your site looked like to Googlebot last Tuesday, or are you guessing from a crawl you ran six months ago? Schedule the next one today, diff it against the last, and watch what changes. That habit, more than any single audit, is what keeps a site healthy as it grows.