How to Crawl a Website for Broken Links

Quick Answer

To crawl a website for broken links, use a website crawler to find failed destinations and identify the pages that link to them. Review HTTP responses, redirect paths, and actual page content before choosing a repair. Then correct the link or destination and run another crawl to confirm the result. Coverage depends on the crawler and its settings.

Key Takeaways

A useful broken-link audit produces repair tasks, not just a count of errors. Each finding needs enough evidence to explain what failed, where the link appears, and what should happen next.

• Record the source page and destination URL together so each failure can be traced to a specific link.
• Decide whether the crawl must check internal links, outbound links, rendered JavaScript content, or several of these.
• Treat a 404 as a finding to investigate, not an automatic instruction to create a redirect.
• Inspect the final destination when a link passes through redirects before reaching its target.
• Recheck timeouts, access restrictions, rate limits, and server errors before classifying a destination as permanently broken.
• Choose between correcting a URL, restoring content, redirecting to a relevant replacement, or retaining a genuine 404 or 410.
• Prioritize links that affect navigation, conversions, and important content instead of using an invented acceptable error percentage.
• Run a follow-up crawl and record which repairs worked, which remain unresolved, and what the crawl could not test.

The central decision is to establish the source, destination, observed response, and intended content state before making changes. That process reduces unnecessary redirects and gives the people responsible for repairs a clear record to follow.

Start With the Right Crawl Scope

Before starting a crawl, decide which parts of the website and which kinds of links need to be checked. A crawl of internal HTML pages may find many problems, but it does not automatically test every outbound destination or every link created by JavaScript.

Start with the site's preferred hostname and a representative starting URL. Decide whether the audit should include relevant subdomains, alternate URL paths, paginated pages, and pages that require a login. Also choose whether PDFs, images, scripts, stylesheets, and other non-HTML resources belong in the report. These choices shape what the crawler can discover and what the results can prove.

Link discovery and link validation are separate tasks. A crawler may extract an external URL without requesting that destination or recording its current HTTP response. Some tools use separate reports or settings for outbound checks. Ahrefs documents its process for checking broken outbound links. Check the current tool instructions before assuming both internal and external checks are enabled.

Rendered content needs attention, too. Google's guidance on crawlable links explains the role of standard anchor elements with an href attribute. JavaScript-rendered pages may expose links that a basic HTML crawl does not see. If important links appear only after scripts run, compare the crawler's output with the rendered page.

Access restrictions also affect coverage. Review robots rules, authentication, request limits, and any known firewall or server restrictions. Do not bypass access controls without permission. For a large site, set a suitable crawl limit and request rate to avoid placing needless pressure on the server.

Record the starting URL, included hostnames, link types, rendering mode, crawl date, and excluded resources. This scope statement helps others judge the report. Without it, a finished crawl can look complete even when major parts of the site were never tested.

Find the Broken Destination and Its Source Page

The most useful report connects each failed destination to the page containing the link. A list of error URLs alone is not enough. You need to know where the link appears before you can decide how to repair it.

A common workflow is to finish the crawl, open the response-code report, filter for client errors, and inspect the inlinks or referring-page details. Inlinks are the links that point to a destination. They can show the source page, anchor text, and sometimes the context around the link.

Screaming Frog's broken-link checking tutorial describes filtering its Response Codes report for Client Error (4XX), examining inlinks, and exporting the results. Other crawlers may use different labels, but the core task is the same: find the failed destination and identify each page that points to it.

Wait for the crawl to finish before treating the report as final. If you inspect results while the crawl is running, mark them as preliminary. One destination may have several source pages, and each source link may need a separate decision. Preserve the observed status, final URL, and anchor text when the crawler provides them.

Consider a hypothetical case in which three pages link to /services/old-package, and that destination returns 404. The right repair might be to update all three source pages, restore the old page, or redirect it to a relevant replacement. The error URL alone cannot tell you which action is appropriate.

Export the findings to a spreadsheet or task system so each repair can be assigned and tracked. At minimum, retain the source URL, destination URL, link type, anchor text, observed response, final destination, and date checked. Add a diagnosis and an owner after investigation. This turns a crawler report into a working repair queue.

A completed crawl does not prove that every link on the site has been tested. The report remains limited by its scope, the pages it could reach, and the destinations it actually requested.

Separate Internal and External Link Checks

Internal and outbound links need different checks because they lead to different destinations and often require different repairs. Internal links point to URLs within the audited website. Outbound links lead to other websites.

An internal-link audit can uncover broken navigation, incorrect paths, removed articles, and links that were not updated after a site change. Since you control the source page and often the destination, you may be able to correct the URL or restore the missing resource directly.

An outbound-link audit checks references to pages on other sites. These might include research citations, supplier pages, official resources, partner websites, or embedded assets. You may not control the destination, so the repair could involve updating the reference, removing it, or finding a dependable replacement.

Ahrefs explains its internal broken-link checks for 4XX destinations. Its separate outbound-link workflow shows why external status checks should be confirmed rather than assumed. Feature names, crawl allowances, and interface steps can change, so check the current vendor instructions before relying on a particular setting.

The comparison below shows what to retain for each audit type. Use it to decide whether your chosen crawl covers the full maintenance task or whether you need a separate external check.

If your maintenance review covers both types, run both checks and label their results separately. That makes it easier to assign internal fixes to your team and route third-party failures to the content owner who can update the reference. It also prevents a report from claiming external coverage that was never tested.

Interpret the Response Before Choosing a Fix

An HTTP status is evidence about a request, not a complete diagnosis of the content. A 404 often means the requested resource was not found, but a 403, 429, or 5XX response can have a different cause. Inspect the page and the final destination before deciding whether a link is genuinely broken.

Google's official guidance on crawling errors explains how missing pages, moved content, and soft 404s need different treatment. A genuinely missing resource can return 404 or 410. A moved page with a clear replacement may use a permanent 301 redirect. A soft 404 can return HTTP 200 even though the page tells visitors that the requested content is missing.

Use the matrix below to choose the next diagnostic step. It combines observed response behavior with practical repair advice. It is a working method for site maintenance, not a complete HTTP specification.

A successful response is not enough if the destination contains the wrong material. Likewise, a non-200 response does not always mean a page has been deleted. Check the final URL after redirects and compare the returned content with the purpose of the original link. The distinction is especially useful when a crawler reports an error that does not appear in a normal browser session.

Choose Between Correcting, Restoring, Redirecting, and Removing

The right repair depends on what the destination is supposed to do. Correct the source link when its URL contains a typo or points to the wrong page. Restore the resource if it was removed by mistake and should still be available. If content moved, use a permanent redirect when a genuinely relevant replacement exists.

A redirect should preserve the user's intent. If an old product page has a direct successor, redirecting to that successor may help visitors. Sending every deleted product, article, or document to the homepage is not an equivalent fix. The homepage may return 200 while leaving visitors without the information they requested.

When content has been intentionally removed and no suitable replacement exists, a genuine 404 or 410 may be appropriate. Update internal links that still point to the removed resource. For an outbound link, find a dependable replacement if the reference still serves a useful purpose. Remove it if the information is no longer needed.

Do not create a redirect simply to make the error count smaller. The goal is to give users and crawlers a sensible route to the intended information, not to hide every missing URL behind a successful response.

Recheck Redirects, Timeouts, and Crawl False Positives

Repeat uncertain checks before classifying a destination as permanently broken. Redirect chains, server errors, timeouts, DNS failures, and access restrictions need separate investigation because their results can change between requests.

A redirect chain occurs when one URL sends the request to another URL, which then sends it somewhere else. The final destination may fail even when the first redirect responds normally. Screaming Frog's guide to broken-link building discusses tracing redirect paths and investigating URLs that return no response. Follow the full chain when the crawler exposes it, then decide whether the source link should point directly to the intended destination.

Crawl results can also vary with changing site content, request timeouts, server errors, robots rules, user-agent differences, and JavaScript rendering. Screaming Frog lists these and related causes in its SEO Spider FAQ. These are possible explanations, not proof of what happened on a particular site.

A 403 response may reflect an access rule. A 429 may mean the server is limiting requests. A 5XX response may be temporary. A timeout may come from the destination, the network, or the crawler's settings. Recheck the URL and, when appropriate, compare the result with a normal browser request. Respect access controls and avoid flooding a remote server with repeated requests.

Keep both observations in the repair record: the first result and the recheck result. If they differ, mark the finding as intermittent or unresolved rather than treating it as a confirmed permanent failure. This helps developers focus on real defects without changing valid links unnecessarily.

Turn Findings Into a Prioritized Repair Queue

Prioritize repairs by their effect on visitors and important site tasks. Broken links in the main navigation, service pages, lead forms, checkout paths, or frequently used resources usually deserve attention before an obsolete reference deep in an old article. This is a practical maintenance rule, not an official Google scoring system.

A useful repair record keeps the evidence and proposed action together. The following worked example is hypothetical. It does not report a real crawl of IndexNowFast or any other website.

This record is more useful than a row that says only “404.” It identifies the affected page, retains the observed evidence, and separates a confirmed response from an unconfirmed explanation. The proposed action remains conditional until the content owner establishes the intended destination.

For your own audit, retain the source URL, destination URL, link type, anchor text, observed status, final URL, redirect chain, first-observed date, recheck result, business importance, diagnosis, recommended action, owner, and verification outcome where available. You do not need every field for every link, but keep enough context to reproduce the decision and explain why the repair was chosen.

Assign one person or team to each repair. Content editors can update references, developers can correct templates or routing, and server administrators can investigate access or availability problems. Track unresolved findings separately so an incomplete repair queue is not mistaken for a clean audit.

There is no universal acceptable percentage of broken links established by the sources reviewed here. Report actual findings and their consequences. A small number of broken links in critical paths may matter more to users than a larger number of obsolete references on rarely visited pages.

Verify the Fix With a Follow-Up Crawl

A repair is complete only when the source page and destination behave as intended. After changing a link, restoring a page, or adding an appropriate redirect, revisit the affected path and run the relevant crawl or request check again. Record the outcome rather than assuming that a saved change solved the problem.

Start with the original source page. Confirm that the link now points to the intended URL and that its anchor text still makes sense. Request the destination, inspect the final URL after any redirect, and confirm that the expected content appears. A 200 response alone does not prove success if the destination shows a soft 404 or unrelated content.

Next, rerun the relevant part of the crawl. For a small site, this may be a full crawl. For a large site, a focused recrawl can help check the changed pages, followed by a broader crawl when appropriate. Use the same scope and comparable settings where possible so the before-and-after results are meaningful.

Close each repair record with a result such as resolved, unresolved, intermittent, or not tested. Include the date and crawl scope. If a failure persists, return it to the queue with the latest evidence. If the response changes but the content remains wrong, record the remaining issue rather than marking it resolved.

Choose the next audit date based on how often the site changes, how important its key user paths are, and how often past checks have found failures. A frequently updated publishing site may need more frequent checks than a small brochure site. The reviewed evidence does not establish one universal audit interval.

Keep Broken-Link Auditing Separate From Google Indexing

A broken-link crawl tests links and their destinations. URL discovery and indexing workflows address different questions. Submitting a URL to an indexing service does not correct a typo, restore a missing page, or repair a link on another page.

A page can be accessible but not indexed. A page can also be indexed while another page links to an incorrect destination. Crawlability, HTTP availability, link discovery, and search inclusion are related, but they are not interchangeable. Google's guidance on crawlable links explains how links can be discovered, while its crawling-error guidance helps diagnose missing or moved resources.

If your actual problem is that Google is not including an accessible page in Search, follow a separate indexing diagnosis. IndexNowFast has an existing article on why Google isn't indexing your pages, which covers related issues such as discovery, rendering, and indexing. That article addresses a different problem from finding broken links. Choosing the right diagnostic path saves time and helps avoid changes that cannot resolve the issue.

IndexNowFast's website describes URL discovery and indexing-related workflows. Its blog resource hub lists related material on indexing, crawlability, and troubleshooting. The public pages reviewed for this article do not establish that the service provides a dedicated full-site broken-link crawler. Do not assume that URL submission replaces a crawler or that submitting a failed destination repairs the underlying link. If you are assessing IndexNowFast's URL discovery and indexing services, its pricing page provides the published commercial details; those details should not be read as proof of a dedicated broken-link auditing feature.

Frequently Asked Questions

How do I find which page contains a broken link?

Open the crawler's report for the failed destination and inspect its inlinks or referring pages. Record the source URL and destination URL together, along with the anchor text and observed response when available. This lets you identify the exact link that needs attention instead of editing the destination without understanding how visitors reach it.

Should I check external links as well as internal links?

Check both if your audit needs to cover both. An internal crawl may find links to failed pages on your own site without checking the current response of every external destination. Confirm that the crawler supports outbound-link status checks and that the relevant check actually ran before reporting external coverage.

Does every 404 need a redirect?

No. Redirect a moved page when a relevant replacement exists. If a resource was intentionally removed and no suitable replacement is available, returning 404 or 410 may be correct. Update source pages that still link to it, and avoid sending unrelated missing URLs to the homepage merely to reduce the error count.

Can a page return 200 and still be broken?

Yes. A soft 404 can return HTTP 200 while showing a missing-page message or otherwise failing to provide the expected content. Check the response and the page itself. If the destination is supposed to serve a specific resource, confirm that the expected material is actually available.

What should I do when a URL times out or returns a server error?

Recheck the destination and investigate possible access, network, rate-limit, DNS, or server problems. A timeout or 5XX response does not by itself prove that the page has been permanently removed. Record the new result and keep the finding unresolved until you have enough evidence to choose a repair.

Will fixing broken links improve rankings?

Accurate links support navigation and help crawlers discover relevant pages, but the reviewed sources do not establish a guaranteed ranking improvement from repairing any particular broken link. Fix broken links because they can obstruct users, lead to missing content, and weaken site maintenance. Do not promise a specific ranking gain or assume every failure causes a ranking penalty.

How often should I crawl a website for broken links?

Set the schedule according to the site's size, update rate, critical user paths, and history of link failures. A site that publishes or changes pages often may need more frequent checks than a site with stable content. There is no universal audit interval supported by the evidence reviewed here.

Can a basic crawler find every broken link?

No single crawl should be assumed to cover every link and failure mode. Authentication, robots rules, JavaScript rendering, server restrictions, timeouts, and external rate limits can affect the results. State the crawl scope, review known gaps, and use targeted checks when important links may be hidden from the standard crawl.

Final Recommendation

Treat a broken-link audit as a repeatable diagnosis-and-repair process, not a one-time count of failed URLs. Start with a clear crawl scope, retain each source page and destination together, and record the response and final URL. Recheck uncertain failures before choosing a repair. Correct wrong links, restore content that should exist, and redirect moved pages only when a relevant replacement is available.

Give priority to links that affect navigation, conversions, and access to important content. Assign an owner to each repair and record the reason for the chosen action. After the change, revisit the source page, test the destination, and run a follow-up crawl with comparable settings. Confirm that the expected content appears rather than relying on an HTTP 200 response alone.

Keep unresolved and intermittent findings visible in the final report. State the crawl date, scope, excluded areas, and checks that were not performed. This lets the next reviewer distinguish confirmed repairs from gaps that still need work. The result is a more reliable maintenance process, with fewer unnecessary redirects and a clearer record of what the audit actually established.

SEO Title: How to Crawl a Website for Broken Links

Meta Description: Learn how to crawl a website for broken links, find source pages, interpret HTTP errors, choose the right fix, and verify repairs.