Crawl errors happen when Google’s bot (Googlebot) tries to access a URL on your website and cannot retrieve it successfully. The page might not exist, the server might not respond, a redirect might loop endlessly, or a robots.txt rule might block access. Whatever the cause, the result is the same: Google cannot read that page, cannot index its content, and cannot show it in search results.
Not every crawl error is a crisis. A 404 on a page you intentionally deleted is working as designed. But crawl errors on pages that should be ranking, pages with backlinks, or pages representing key services are actively costing you visibility. The skill is knowing which errors matter and which you can safely ignore.
Where Do You Find Crawl Errors in Google Search Console?
Google retired the old “Crawl Errors” report years ago. The modern equivalent is the Page Indexing report (found under Indexing in the left navigation of Google Search Console). This report shows every URL Google has attempted to crawl and its current status.
The report organizes URLs into two buckets:
- Indexed pages: URLs that Google successfully crawled and added to its search index
- Not indexed pages: URLs that Google attempted but did not index, along with the specific reason why
Within the “not indexed” section, each URL gets a status label explaining why it was excluded. The most actionable ones for crawl error diagnosis:
| Status | What It Means | Action Required |
|---|---|---|
| Not found (404) | Page does not exist at this URL | Redirect to relevant page or confirm intentional deletion |
| Server error (5xx) | Your server failed to respond | Investigate server logs, fix hosting/application issues |
| Redirect error | Redirect chain is broken or loops | Fix redirect configuration |
| Blocked by robots.txt | Your robots.txt prevents crawling | Update robots.txt if page should be crawlable |
| Crawled, currently not indexed | Google crawled it but chose not to index | Improve content quality or consolidate with stronger page |
| Discovered, currently not indexed | Google knows it exists but has not crawled it yet | May need more internal links or sitemap inclusion |
Click any status to see the specific affected URLs. This is where diagnosis begins.
Which Crawl Errors Actually Need Fixing?
The Page Indexing report often shows hundreds or thousands of “not indexed” URLs. Many of these are intentional (noindexed pages, paginated archives, parameter URLs). Fixing everything indiscriminately wastes time.
Fix immediately if the URL:
- Is a page you want ranking in search results
- Has external backlinks pointing to it (check your backlink data)
- Was previously receiving organic traffic
- Is an important service, product, or content page
- Represents a pattern (a template error affecting many pages)
Ignore safely if the URL:
- Is a page you intentionally removed or noindexed
- Is a duplicate caught by canonical tags (working as intended)
- Is a parameter URL or filter page that should not be indexed
- Is a staging or development URL that leaked into the index
- Is an old campaign landing page with no links and no traffic
The key question for each error: “Would fixing this result in a useful, rankable page appearing in search results?” If yes, fix it. If not, leave it alone.
How Do You Fix 404 Errors?
A 404 means the URL returns a “not found” response. This happens when pages are deleted, URLs are changed without redirects, or internal links point to URLs that never existed (typos, broken CMS output).
Step 1: Determine if the page should exist.
If you intentionally deleted this content and it has no backlinks or traffic history, the 404 is correct. No action needed.
Step 2: If the content moved, redirect it.
Set up a 301 redirect from the old URL to the new location. This tells Google the content permanently moved and transfers any link equity from the old URL to the new one.
Step 3: If the content no longer exists anywhere on your site, redirect to the closest relevant page.
Do not redirect everything to the homepage. Google treats mass redirects to the homepage as soft 404s. Instead, find the most topically-similar page on your site and redirect there.
Step 4: Fix internal links pointing to the broken URL.
Even after setting up a redirect, update internal links throughout your site to point to the correct destination directly. This avoids unnecessary redirect hops and passes link equity more efficiently.
Step 5: Fix external sources if possible.
If you control the linking source (your social profiles, directory listings, partner sites you have relationships with), update those links to the correct URL.
How Do You Fix Server Errors (5xx)?
Server errors indicate your hosting environment failed to deliver the page. This could be a temporary overload, an application crash, a misconfigured server, or a resource limit being hit.
5xx errors are more urgent than 404s because they signal infrastructure problems that may affect your entire site’s crawlability. If Googlebot encounters repeated server errors, it reduces crawl rate to avoid overloading your server, which means less of your content gets discovered and indexed.
Common causes and fixes:
503 (Service Unavailable): Your server is overloaded or in maintenance mode. If this happens during traffic spikes, your hosting plan cannot handle the load. Upgrade hosting resources or implement server-level caching to reduce load.
500 (Internal Server Error): A catch-all for application crashes. Check your server error logs for the specific PHP error, database connection failure, or plugin conflict causing it. On WordPress, this is often a plugin conflict or a memory limit exceeded.
502 (Bad Gateway): Your server’s proxy or load balancer cannot reach the backend application. This is typically a hosting infrastructure issue. Contact your hosting provider if persistent.
504 (Gateway Timeout): The backend took too long to respond. Usually caused by slow database queries, unoptimized PHP scripts, or insufficient server resources. Optimize the slow query or increase server timeout limits.
If server errors appear intermittently (sometimes the page loads, sometimes it errors), the issue is likely resource-related. Your server works fine under low load but fails under moderate to high load. Caching and hosting upgrades are the typical solutions.
How Do You Fix Redirect Errors?
Redirect errors occur when Google follows a redirect but encounters a problem: the redirect loops back to itself, creates a chain too long to follow, or redirects to a page that also returns an error.
Redirect loops: URL A redirects to URL B, which redirects back to URL A. Google gives up and reports an error. Fix by identifying the circular reference in your redirect rules and pointing one of them to the correct final destination.
Long redirect chains: URL A redirects to B, which redirects to C, which redirects to D. While Google will follow up to about 10 hops, each hop adds latency and wastes crawl budget. Flatten the chain so A redirects directly to D.
Redirects to error pages: A redirect pointing to a URL that returns a 404 or 500. The redirect itself works but the destination is broken. Fix the destination URL or update the redirect target.
How to audit redirects: Run a crawl of your site and look for any URL that returns a 3xx status code. Check what it redirects to and verify that the final destination returns a 200 status. Flag chains longer than one hop and any redirects landing on non-200 pages.
How Do You Handle Robots.txt Blocking Issues?
Your robots.txt file tells search engines which parts of your site they are allowed to crawl. If important pages are accidentally blocked, Google will never index them regardless of how well they are optimized.
Check your robots.txt (yourdomain.com/robots.txt) for overly broad rules. Common mistakes:
- Blocking entire directories that contain important content (e.g.,
Disallow: /blog/when you want blog posts indexed) - Blocking CSS and JavaScript files that Google needs to render your pages properly
- Blocking parameter URLs with rules so broad they catch real pages too
- Left-over rules from development or staging that were never removed for production
To test: use the URL Inspection tool in Search Console. Enter any URL and check if it reports “blocked by robots.txt.” If it does, you know robots.txt is the problem for that specific URL.
The fix is editing robots.txt to remove or narrow the offending rule. After updating, use Search Console’s robots.txt tester to verify your changes work as expected before waiting for Google to re-crawl.
What Is the “Crawled but Not Indexed” Problem?
This status means Google successfully crawled the page but decided not to add it to its index. The page is accessible, it loaded properly, but Google determined it was not worth indexing. This is not a technical error. It is a quality judgment.
Common reasons Google chooses not to index a crawled page:
- Thin content: The page does not provide enough unique value (too short, too generic, or largely duplicating content found elsewhere on your site or the web)
- Low perceived authority: The page has no internal links pointing to it, no external links, and lives deep in the site architecture
- Duplicate content: The content is too similar to another page Google has already indexed (even without an explicit canonical tag)
- Low demand: Google has determined that no one is searching for what this page covers
Fixes vary by cause:
- Add substantial, unique content if the page is thin
- Build internal links to the page from related, authoritative pages on your site
- Consolidate near-duplicate pages into one comprehensive page
- Ensure the page appears in your XML sitemap
- Add the page to your site’s navigation or hub pages so it gets discovered more quickly
Sometimes the right answer is to accept that a page does not deserve indexation. Not every URL on your site needs to rank. Utility pages, tag archives, thin category pages, and author pages often do not merit indexation and their exclusion is correct behavior.
What Priority Order Should You Fix Crawl Errors In?
When facing dozens or hundreds of errors, prioritize by business impact:
| Priority | Error Type | Reasoning |
|---|---|---|
| 1 | Server errors on high-traffic pages | Active revenue loss |
| 2 | 404s on pages with backlinks | Losing link equity every day |
| 3 | Blocked important pages (robots.txt) | Pages that could be ranking are invisible |
| 4 | Template-level errors (fix one pattern, resolve many URLs) | High efficiency per fix |
| 5 | 404s on pages with historical traffic | Recovering lost visibility |
| 6 | Redirect chains on important pages | Crawl efficiency and link equity |
| 7 | Crawled-not-indexed on strategic content | Content investment not paying off |
After fixing each batch, click “Validate Fix” in Search Console for the relevant status. This tells Google to re-check the affected URLs and confirms the fix worked. Monitor the validation progress over the following one to two weeks.
Set a recurring monthly check of your Page Indexing report. New crawl errors will appear over time as your site changes, links break, and content is updated. Catching them monthly prevents small issues from compounding into large indexation gaps. The sites with the cleanest indexation profiles are not the ones that never have errors. They are the ones that identify and fix errors quickly before they accumulate.