Find and fix crawl errors in Search Console

Until 2019, Search Console had its own Crawl Errors report. Anyone who opened it for a new website often saw thousands of entries with 404s, soft 404s, server errors and access denied. The report is gone, but the errors remain. Today they live in three places.

This guide shows where to find crawl errors today, which ones really matter and how to work through them. Why pages stay out of the index even when crawling works is covered in the guide Google indexing.

Where crawl errors are today

Google switched off the old report at the end of March 2019 together with the old Search Console, and the Crawl Errors API with it. According to Google, there is no replacement for the API. The information is now spread like this:

• Crawl stats (Settings → Crawl stats) shows how Googlebot experiences your server, with host status, responses by status, file types, purpose and Googlebot type.
• Page indexing (Indexing → Pages) shows which URLs are not in the index and why, including every reason related to crawling.
• URL Inspection (search field at the top) shows a single URL, with a live test.

From the old report to today

The table shows where the categories of the old report can be found today.

The Crawl stats report

You find the report under Settings → Crawl stats → Open report. It is only available for root-level properties, meaning Domain properties or URL-prefix properties such as https://www.example.com. Google aims it at advanced users and says that sites with fewer than 1,000 pages usually do not need it. For large websites it is the most important source of crawl errors.

Host status for robots.txt, DNS and server connectivity

Host status combines three checks. This is where the former site errors end up.

• robots.txt fetch shows how often fetching robots.txt failed.
• DNS resolution shows whether your DNS server did not recognize the host name or did not respond.
• Server connectivity shows whether your server did not respond or did not deliver a complete response.

Each check has a chart with a red threshold line. If a day goes above it, Google counts it as a problem. The status has three levels. Either there were no significant problems in the last 90 days, or there were problems in the last 90 days but not in the last week, or there was at least one problem in the last week.

Crawl responses by status

This breakdown shows the share of each response among all crawl requests, not per URL. Google sorts the responses into three groups.

File type, purpose and Googlebot type

• By file type the report separates HTML, image, JavaScript, CSS, PDF, JSON and more. The type “Unknown (failed)” counts requests that got no response at all. If many JavaScript or CSS files fail, Google cannot fully render your pages.
• By purpose “Discovery” stands for URLs Google crawls for the first time and “Refresh” for known URLs. After a migration the share of discovery goes up.
• By Googlebot type you see smartphone, desktop, image, video, page resource load, AdsBot, StoreBot and others. This tells you which crawler runs into errors.

At the top of the report you also find total crawl requests, total download size and average response time. A rising response time often signals server problems before errors show up.

Crawl-related reasons in Page indexing

The Indexing → Pages report (formerly “Coverage”) lists every reason under “Why pages aren’t indexed”. The following ones are directly related to crawling. The other reasons, such as “Duplicate” or “Excluded by ‘noindex’ tag”, are covered in the guide Google indexing.

The report shows up to 1,000 example URLs per reason. On large websites many affected URLs are therefore missing. Once you have fixed every case of a reason, click Validate fix in the detail view. According to Google, validation usually takes about two weeks, sometimes longer. This replaces the former “Mark as fixed”.

URL Inspection for single URLs

URL Inspection shows you, for a single URL, what Google saw during its last crawl. Enter the URL in the search field at the top of Search Console.

• View crawled page shows the HTML, a screenshot and the HTTP response from the last crawl.
• Page fetch, Crawl allowed?, Indexing allowed? and Last crawl show whether and when Google could fetch the URL. “Crawled as” names the crawler.
• Referring page and Sitemaps show where Google knows the URL from. This is the successor of the old “Linked from” tab, but it only gives examples.
• Test live URL fetches the URL right now. This is how you check whether a fix works.
• Request indexing is for after a fix. There is a daily limit per property.

A successful live test does not mean the URL will be indexed. Among other things, the live test does not check for duplicates, canonical selection or manual actions.

404 errors from URLs in scripts

Sometimes Search Console reports 404 errors for URLs that are not linked on any page. Often they sit in JavaScript as relative paths, for example in a configuration object or a template string. Google also finds links in embedded code and tries to crawl such paths.

According to Google, such 404 errors do not harm your website. Individual URLs returning 404 do not affect how the rest of your URLs are crawled and ranked. You can ignore them. The only thing that matters is that your server really returns 404 for URLs that do not exist, and no empty page with status 200.

How do you work through a lot of crawl errors?

Lots of errors at once are normal, especially on a website you have just taken over. Most of them are harmless. This order has proven itself:

1. Host status first. Problems with robots.txt, DNS or server connectivity affect the whole website. They take priority over every URL error.
2. Then server errors (5xx). Single errors during maintenance or traffic peaks are normal. If they repeat, Google slows down crawling and visitors see the same errors. Send your developers or hosting provider the example URLs with dates and compare them with the server logs.
3. Then internal causes. If 404s, soft 404s or redirect errors come from your own pages or the sitemap, fix the link or the sitemap entry. Also redirect to the right target so the old URL drops out of the index.
4. Then external links. If a link on another website points to a wrong URL, redirect it with a 301 to the matching target. Asking the site owner for a correction is only worth it for important links.
5. Leave the rest. 404s for deleted pages without links, URLs from scripts and errors whose origin cannot be traced are not a problem. If the same URL keeps coming back, look into it.

It works best to check the reports once a week at a fixed time. Then the new errors stay manageable and you spot outliers.

What Search Console does not show

• Delay. The reports show what Google has already crawled. An error on a rarely crawled page shows up there only after days or weeks.
• Samples. There are at most 1,000 example URLs per reason, and “Referring page” gives only one example.
• Google only. How Bing, AI crawlers or visitors experience your website is not in there.

For a complete list of all requests and status codes you need the server logs, see log file analysis. How to assess and fix 404 errors from every source is explained in the guide Finding and fixing 404 errors.

Frequently asked questions