How to Fix Crawl Errors Found in a Site Audit (Step-by-Step)

Crawl errors are one of the most common problems a full site audit surfaces, and they're also one of the most consequential. If Google can't crawl a page, it can't index it. If it can't index it, that page will never rank, no matter how good your content or backlinks are. So when your audit report lights up with 404s, server errors, redirect chains, and blocked resources, it's worth understanding exactly what each one means and how to resolve it.

This guide walks you through the process step by step: how to read the errors, prioritize them, fix the most common types, and confirm the fixes actually worked. The goal is to restore healthy crawling and indexing so your pages get the visibility they deserve.

What Crawl Errors Actually Are

A crawl error happens when a search engine bot tries to reach a URL on your site and something goes wrong. Google's crawler follows links, reads your sitemap, and revisits pages it already knows about. When it requests a page, your server responds with a status code. Codes in the 200 range mean success. Codes in the 300s are redirects. Codes in the 400s mean the page can't be found or accessed, and 500s mean your server failed to respond properly.

Crawl errors generally fall into two buckets: site-level errors that affect the whole domain (like DNS failures or a server that's down), and URL-level errors that affect individual pages (like a broken link pointing to a page that no longer exists). Site-level errors are rarer but more urgent because they can block crawling entirely. URL-level errors are far more common and usually accumulate quietly over time as content is moved, renamed, or deleted.

If you want a broader refresher on how these fit into a wider health check, our [complete guide to auditing your site for SEO](/blog/website-audit-seo-guide) covers the full landscape before you zoom into crawl issues specifically.

Step 1: Gather Your Crawl Data

Before you fix anything, you need an accurate, complete picture of what's broken. Two sources matter most.

First, Google Search Console. The Pages report (formerly Coverage) shows which URLs Google tried to index and why some failed. It groups issues into categories like "Not found (404)," "Server error (5xx)," "Redirect error," and "Blocked by robots.txt."

Second, a full crawl of your own site. Search Console only shows what Google has encountered; a dedicated crawler visits every internal link and reports issues in real time, including ones Google hasn't gotten to yet. Running a full site audit gives you a consolidated list of broken URLs, their status codes, and crucially, where each broken link is being referenced from.

Export both lists. You want the URL, the status code, and the source page that links to the problem URL. Without the source, you'll fix the destination but leave the broken link in place.

Step 2: Prioritize by Impact

Not every crawl error deserves the same urgency. Sort your list by potential impact so you spend time where it matters.

• Server errors (5xx) come first. These often signal hosting or configuration problems that can affect many pages at once and even hurt crawling of healthy pages.
• Broken pages that used to rank or receive traffic come next. A 404 on a page that earns clicks or holds backlinks is actively costing you.
• Redirect chains and loops follow, since they waste crawl budget and dilute link equity.
• Soft 404s (pages that return a 200 status but display a "not found" message or have no real content) are easy to overlook but confuse indexing.
• Low-value 404s on old, orphaned pages nobody links to can be handled last.

A good rule: cross-reference broken URLs against your analytics and backlink data. A 404 receiving zero traffic and zero links is low priority. A 404 with inbound links from other sites should be fixed today.

Step 3: Fix 404 (Not Found) Errors

The 404 is the crawl error you'll see most often. It means the URL was requested but doesn't exist. You have three main options, and the right one depends on the situation.

Restore the page if it was deleted by mistake or still serves a purpose. This is the cleanest fix when the content should exist.

Redirect the URL with a 301 (permanent) redirect to the most relevant live page when the content moved or was replaced. Point it to a genuinely equivalent page, not just your homepage. Redirecting everything to the homepage is treated by Google as a soft 404 and helps no one.

Leave it as a 404 (or use a 410) if the page is genuinely gone and has no replacement. A 404 is not inherently bad; it's the correct answer for content that no longer exists. Just make sure nothing important still links to it. Use a 410 (Gone) if you want to signal the removal is permanent and intentional.

Whatever you choose, go back to the source page from your export and update or remove the broken internal link. Fixing the destination without fixing the link means bots keep hitting the error.

Step 4: Resolve Server Errors (5xx)

Server errors usually aren't an SEO problem at their root; they're a hosting or application problem with SEO consequences. Common causes include an overloaded server, a plugin or script timing out, database connection failures, or aggressive crawling hitting resource limits.

Start by checking whether the error is intermittent or constant. Reload the affected URLs yourself. If they load fine now, the server may have been temporarily overwhelmed, in which case you should look at hosting capacity, caching, and whether crawl rate needs adjusting. If they consistently fail, check server logs, disable recently added plugins one at a time, and contact your host. Because 5xx errors can suppress crawling site-wide, resolve these before anything else.

Step 5: Clean Up Redirects

Redirects aren't errors themselves, but broken and inefficient redirects create real problems. Look for two patterns.

Redirect chains happen when URL A redirects to B, which redirects to C. Each hop wastes crawl budget and slightly weakens link equity. Flatten chains so the original URL points directly to the final destination in a single step.

Redirect loops occur when A points to B and B points back to A, trapping the crawler. These block indexing entirely and must be broken immediately by correcting one side of the loop.

Step 6: Check robots.txt and Blocked Resources

Sometimes a page isn't broken at all; you've accidentally told Google not to crawl it. Your robots.txt file can block entire directories, and a misplaced rule can hide important sections of your site. Review any "Blocked by robots.txt" entries in your audit and confirm each block is intentional. Also make sure you're not blocking CSS and JavaScript files Google needs to render pages properly.

While you're here, distinguish crawl blocking from indexing rules. A noindex meta tag is a separate control from robots.txt, and confusing the two is a frequent cause of pages vanishing from search. If you're unsure exactly what your audit inspects across these areas, our plain-English explanation of [what a full site audit checks for](/blog/what-does-site-audit-check) breaks down each category.

Step 7: Verify and Monitor

Fixing is only half the job. After making changes, request validation in Search Console for the affected error groups so Google recrawls them. Then re-run your own crawl to confirm the status codes have changed and no new broken links appeared as a side effect of your fixes.

Crawl health isn't a one-time task. New errors appear whenever you publish, redesign, migrate, or delete content. Schedule a recurring crawl so problems get caught while they're small. Pairing this with a routine review of on-page elements, using something like an [on-page SEO audit checklist](/blog/on-page-seo-audit-checklist), keeps both your technical foundation and your content quality in good shape.

Bringing It All Together

Fixing crawl errors comes down to a repeatable loop: gather accurate data, prioritize by impact, apply the correct fix for each error type, and verify the results. Server errors and redirect loops demand immediate attention because they can block crawling broadly. The everyday 404s are less dramatic but add up, so handle the ones with traffic and links first and clean the rest steadily.

Done consistently, this work keeps your most important pages crawlable and indexable, which is the baseline requirement for ranking at all. If you'd like a consolidated, prioritized list of every crawl issue on your site along with the exact pages linking to each broken URL, run a Full Site Audit and start working through the report from the top. You can explore the rest of the toolkit anytime on the [Sweto platform](https://swetofix.com).