How to Crawl a Website for All URLs

Quick Answer

To crawl a website for all URLs, combine sitemap discovery, an internal-link crawl, and available CMS exports or other independent URL sources. Merge and deduplicate the results, then check response codes, redirects, canonicals, and crawl restrictions. No single method guarantees every URL has been found, so define completeness against your sources and crawl settings.

Key Takeaways

A reliable website URL inventory records where each URL came from and what checks it passed. These findings help you build a useful list without mistaking a large export for a complete or accurate one.

• Sitemap files reveal declared URLs, but they may omit pages that exist elsewhere on the site.
• Google sitemap files support up to 50,000 URLs or 50 MB uncompressed per file.
• Internal-link crawls find reachable pages but can miss orphan URLs and inaccessible sections.
• CMS exports, server logs, analytics, and known URL lists can reveal candidates that other sources miss.
• URL normalization must preserve meaningful paths and query parameters while reducing true duplicates.
• HTTP status, redirects, canonical tags, robots directives, and index status describe different URL states.
• IndexNow sends notifications about known URL changes; it does not crawl a website to discover unknown URLs.
• A defensible completeness report documents its sources, crawl settings, exclusions, validation date, and unresolved gaps.

The strongest inventory combines independent sources, preserves the evidence behind each record, and classifies differences instead of discarding them. Its completeness claim should describe what was checked, not assume that every URL on the server has been found.

Define What “All URLs” Means Before You Start

Before crawling, define which URLs belong in the inventory. The answer depends on the site's structure, your access, and the task you need to complete. A public SEO audit may focus on HTML pages. A migration audit may also need PDFs, images, redirect sources, and old URL paths.

Set these boundaries before collecting data:

• Host and protocol: Decide whether to include one hostname, subdomains, HTTP variants, and HTTPS URLs.
• Resource types: Specify whether the inventory covers HTML pages, PDFs, images, downloads, and other files.
• Query parameters: Decide how to handle tracking codes, filters, sorting, pagination, and parameters that change page content.
• Access: Record whether authenticated areas are included. Confirm that you have permission to crawl them.
• Crawl rules: Define seed URLs, robots handling, depth limits, excluded paths, redirect handling, and rendering settings.
• Output goal: Decide whether you need all discovered URLs, public pages, canonical candidates, migration mappings, or a separate list of indexable pages.

These choices affect the final count. A crawl that includes parameter variants will often return more URLs than one that removes them. Neither count is automatically more accurate. Each may answer a different question.

A site owner preparing a redesign may need every old URL that could redirect to a new page. An SEO specialist checking indexable content may instead need a filtered list of live canonical candidates. Keep the broader inventory when possible, then create purpose-specific views from it.

Document the boundary before starting. Otherwise, changes in settings can look like missing pages or sudden growth when you compare later crawls. Also record any rate limits and exclusions. A larger crawl is not better if it crosses access boundaries or overwhelms the site.

Start With the Sitemap, Then Crawl Internal Links

A sitemap shows which URLs the site declares, while an internal-link crawl shows which URLs a crawler can reach from its configured starting pages. Use both methods, compare their results, and retain each source separately. Their differences can reveal coverage gaps, stale entries, and useful URLs that need further checks.

Find the Sitemap and Sitemap Index

Start with the site's robots.txt file and look for a Sitemap directive. You can also check likely sitemap locations and the site's CMS settings. Some systems generate sitemap files automatically. Others build them from a database or publishing workflow.

Google's official sitemap guidance sets two limits for an individual sitemap file: 50 MB uncompressed or 50,000 URLs. Larger sites can split their URLs across several sitemap files and use a sitemap index to point to them.

Do not assume that /sitemap.xml is the only file you need. An index may point to several child files covering different content types or sections. Parse every available child sitemap and record which file supplied each URL. Use absolute URLs and retain the original values for later comparison.

A sitemap is a declaration, not proof that each URL works. It can contain stale entries, omit orphan pages, or list preferred canonical URLs rather than every URL a crawler might encounter. Google treats sitemap submission as a hint, not a guarantee of crawling or indexing.

During the research for this article, the available retrieval tool could not access the conventional robots.txt, sitemap.xml, or sitemap_index.xml endpoints on indexnowfast.com. That result does not establish that those files are missing. A direct technical check is needed to determine their current availability and contents.

Run an In-Scope Internal-Link Crawl

Start a crawler at the homepage and add other useful entry points. Include navigation pages, category hubs, and important sections that may not link directly from the homepage. Follow internal links within the scope you defined.

The resulting list shows which URLs are reachable from those seeds under the crawler's settings. It does not show every URL that exists. An orphan page with no incoming internal links may remain undiscovered. A blocked section, inaccessible page, or link the chosen crawler cannot extract may also be missed.

Crawl settings influence the result. Depth limits can exclude deeply nested pages. URL exclusions may remove useful paths. Redirect rules can change whether the report records an original URL, its destination, or both. Rendering support also varies by crawler, so check the selected tool's documentation if the site relies on JavaScript to create links.

Where available, retain the referring page and crawl depth for each discovery. These fields help explain why a URL appeared and how it was reached. If an important page is absent, inspect its expected parent page and confirm that the crawler could access and parse its links.

IndexNowFast's guide to Google indexing problems discusses related issues, including internal discovery, robots directives, canonical signals, and rendering. Use it when investigating a specific discovery or indexing problem, but keep this task focused on building the URL inventory.

Reconcile Independent URL Sources to Find Missing Pages

The best way to expand a URL inventory is to compare the sitemap and internal crawl with independent records. Where you have authorized access, add CMS or database exports, server logs, analytics landing pages, and existing URL lists. Keep the discovery source attached to every URL.

These sources answer different questions. A CMS export can reveal published routes that are not linked internally. It may also contain drafts, deleted records, stale slugs, or private routes. Server logs show URLs requested during the retained period, but cannot reveal pages that nobody requested during that period. Analytics show measured visits, not every page that exists.

Search-engine reports have their own limits. They describe URLs known to a particular search engine under the available property and reporting rules. Search operators can help find examples, but search results are not an exhaustive database of a site's URL space.

The following coverage matrix shows how the sources complement one another. No row provides a complete inventory by itself. Access also varies by site, so use the sources available to you and document any gaps.

The most useful result is not simply the source with the highest count. It is the combined inventory with clear provenance and explainable differences. A URL found only in a CMS export deserves a different review from one found in both the sitemap and the internal crawl.

Investigate Orphan-Page Candidates

An orphan-page candidate is a URL found in an independent source but not reached by the configured internal-link crawl. It is a candidate for investigation, not automatic proof of a technical fault. The page may be intentionally unlinked, unpublished, restricted, or obsolete.

Compare its publication state, response status, internal links, and sitemap membership. If the page should be public and discoverable, review the linking structure. If it should not be public, determine whether the URL should remain accessible and whether its current status matches the site's intended behavior.

Do not claim to have found orphan pages until an actual crawl and source comparison identify candidates. This workflow explains how to find them, but it supplies no measured counts for a site that has not been crawled.

Normalize and Deduplicate Without Losing Meaningful URLs

Before comparing sources, create a consistent URL value for matching records. Keep the original URL as well. Normalization helps group genuine duplicates, but an aggressive rule can erase important differences between pages.

URLs may differ in protocol, hostname, trailing slash, letter case, percent-encoding, fragments, or query parameters. Some differences are cosmetic for a particular site. Others change the requested resource or page content. The correct rules depend on the application's URL behavior.

A safe workflow preserves three things: the original discovered URL, a normalized comparison value, and the reason for any transformation or exclusion. Remove tracking parameters only when you can establish that they do not affect the resource being inventoried. Do not remove filters, pagination values, or other parameters blindly.

Redirects and canonical declarations also need separate fields. A redirect shows where a request is sent. A canonical tag expresses a preferred URL for a page. Neither should erase the original record before you have saved the evidence.

A Worked Example of Source Reconciliation

Consider a hypothetical store whose sitemap lists https://example.com/products/widget, while an internal link points to https://example.com/products/widget?ref=nav. These are illustrative URLs, not observed results from a real crawl.

Retain both original values until the site's behavior is checked. Record each discovery source, the response status, the final URL after redirects, and the canonical declaration if one is present. Then determine whether the parameter changes the page, serves only as tracking, or produces a distinct resource.

If the parameter has no meaningful effect, the inventory may group both records for reporting while preserving the original URL and its discovery history. If it changes the content, removing it during normalization could hide a real page variant. A count difference alone cannot settle that question.

The same method applies to alternate hosts, trailing-slash variants, and redirect sources. Classify the evidence first, then decide which URLs belong in the final view for your audit.

Validate Every Candidate URL and Classify Its State

After discovery and deduplication, validate each URL using the checks your tools support. At a minimum, collect response status, redirect destination, and content type where available. Also inspect canonical declarations, robots directives, and fetch errors when the workflow supports them.

Keep these results separate. A URL may be discovered but not fetched successfully. It may return a successful response but carry a noindex directive. It may be eligible for indexing yet absent from Google's index. Each state calls for a different next step.

Separate Discovery, Fetching, Indexability, and Indexing

Use explicit fields or report categories for each state:

• Discovered: A sitemap, link, export, log, or other source supplied the URL.
• Fetched: A crawler or inspection tool requested the URL and recorded a response or fetch error.
• Indexability signals: The response, robots directives, canonical declaration, and other relevant page signals have been checked.
• Indexed: A separate search-engine report or inspection indicates that the URL is in the relevant index at the recorded time.

A successful HTTP response does not prove that a URL is indexed. A crawler's status report also cannot establish Google's index state. Google's recrawl guidance explains that recrawl requests do not guarantee inclusion in search results. Sitemap submission and individual URL Inspection serve different purposes.

Robots.txt controls crawler access. It is not a substitute for a noindex directive. A crawler blocked by robots rules may be unable to inspect the page's content and meta robots directive. Record what was actually observed and mark unchecked fields as unknown rather than assuming that the page is indexable.

Canonical tags require care, too. A page can declare a preferred canonical URL, but the declaration alone does not prove that Google selected it. Compare the requested URL, canonical declaration, and redirect behavior before deciding how to classify the record.

For an individual URL that needs further Google-specific investigation, consult IndexNowFast's Google URL Inspection Live Test guide. It provides a separate diagnostic path; it does not replace the site-wide inventory.

Classify Redirects, Errors, Blocked URLs, and Duplicates

Do not discard a URL simply because it is not a live canonical page. Its role may still matter to a migration, redirect audit, or discovery investigation. Preserve the original evidence and classify each record according to the inventory's purpose.

The classification table below provides a practical review framework. Assign a category only when the available evidence supports it, and leave uncertain cases for manual review rather than forcing a conclusion.

Differences between sitemap and crawl counts are useful diagnostic leads. A sitemap may list preferred canonical URLs, while a crawler also finds redirect sources, query variants, and linked resources. Review the records behind the difference before changing the site or excluding URLs.

The final classification should reflect the goal. A redirect source may be excluded from a list of live canonical pages but remain essential to a migration inventory. A PDF may not be an HTML page, yet it could be a key resource that needs to remain accessible.

Build an Export That Supports Audits and Repeat Crawls

Export one record per retained URL, with enough context to explain where it came from and what happened during validation. A CSV is useful because it can be filtered, joined with other reports, and compared across crawl dates. Some fields may come from different tools rather than one crawler export.

Use a schema like the one below. These are recommended field names, not a claim that any specific crawler supplies every value natively. The purpose is to preserve enough evidence for another person to reproduce decisions and investigate exceptions.

Not every source will populate every field. A sitemap parser may know sitemap membership but not the current response status. A server-log export may show a request but not the page's canonical declaration. Join reports where needed, and leave unavailable or unchecked values clearly marked.

Retain the crawl configuration, source files, normalization rules, exclusions, and timestamps with the CSV. These records make it possible to reproduce a decision and explain why two reports differ. Without them, a URL count has little value as an audit record.

Measure Coverage and Decide When the Crawl Is Finished

A crawl is finished when it has met its documented scope and its remaining gaps have been reviewed or reported. No universal URL count or coverage percentage proves that every possible URL has been found. The total number of URLs across a site's CMS, server, logs, and search engines may be unknown.

Report measurable results instead of claiming universal completeness. Useful measures include:

• Unique URLs supplied by each discovery source.
• URLs found in more than one source.
• Additional candidates contributed by each source after deduplication.
• Response errors, redirects, blocked URLs, and excluded variants.
• Orphan-page candidates and records requiring manual review.
• New, changed, and no-longer-seen URLs compared with a prior inventory.

These measures help explain where coverage is strong and where uncertainty remains. If the CMS export adds many URLs beyond the internal crawl, investigate their publication state and reachability. If the sitemap and crawler differ, review the underlying records rather than treating the larger number as automatically correct.

Repeat the same workflow after major releases, migrations, or URL-structure changes. For sites that publish often, use a schedule that reflects their publishing rate and operational needs. Save each run separately and compare the results. A stable inventory across repeat crawls increases confidence in the configured workflow, but it still cannot prove that no unrecorded URL exists.

A defensible completion statement should identify the host and resource scope, the sources checked, the crawl settings, the validation date, the exclusions, and the unresolved gaps. For example, say that the inventory covers URLs found in the listed sitemaps, the configured internal crawl, and the available CMS export. Do not claim that it contains every URL on the server unless independent evidence supports that broader statement.

Use Indexing and Notification Tools Only After Discovery

Once the inventory is built, use indexing tools for their intended purpose. URL discovery, technical crawling, index-status checks, and URL-change notifications are separate operations. Combining them can create a useful workflow, but one does not replace the others.

Google sitemap submission can help communicate many URLs to Google. URL Inspection is better suited to investigating an individual URL in an owned property. Neither method guarantees that a page will be indexed. Google's recrawl guidance explains the distinction between requesting a recrawl and gaining inclusion in search results.

The official IndexNow documentation describes a protocol for notifying participating search engines about known URL additions, changes, or deletions. It supports individual notifications and batches of up to 10,000 URLs per POST request. That limit concerns notification batches, not the number of URLs a crawler can discover. A successful notification confirms receipt of the notification, not indexing.

IndexNowFast's Google Index Checker page describes bulk index checks of up to 10,000 URLs and CSV export. These are vendor-stated product capabilities, not independent performance findings. Check the current product page before relying on those limits. An index check is a point-in-time observation, not a permanent property of a URL.

A sensible sequence is to build and validate the inventory first, select URLs that need index-state investigation, and notify search engines about known changes when appropriate. Do not submit a large list simply because it exists. First confirm that the URLs are relevant, authorized, and correctly classified.

Repeat the Inventory After Important Site Changes

A URL inventory is a snapshot, not a permanent record of the site. Re-run the same workflow after migrations, major releases, navigation changes, or URL restructuring. Choose any routine interval based on publishing frequency, site size, and the cost of missing a change.

Keep the previous CSV, source files, crawl configuration, and exclusions. Compare the new results against that baseline to identify newly discovered URLs, removed records, changed status codes, new redirect destinations, and changes in canonical declarations. Review the differences before replacing the prior inventory.

A new URL is not automatically a problem. It may reflect a planned launch, a new content section, or a useful discovery source. A missing URL also needs context. It could have been removed on purpose, become inaccessible, or fallen outside the new crawl configuration. The comparison gives you a review queue, not an automatic verdict.

If the site changes frequently, a repeatable process is more useful than an occasional one-off export. Preserve the previous inventory so teams can distinguish genuine site changes from changes caused by crawl settings or source availability. Record the reason for important exclusions and retain unresolved cases until they can be reviewed.

Frequently Asked Questions

Can I find every URL using only a sitemap?

No. A sitemap lists URLs supplied by the site owner or its publishing system. It can omit pages, stale paths, and orphan URLs. Compare it with an internal-link crawl and other authorized sources.

How do I find pages missing from the sitemap?

Run an internal-link crawl and compare its results with every available sitemap file. Then investigate additional candidates from CMS exports, server logs, analytics, and known URL lists. Validate each candidate before deciding whether it belongs in the final inventory.

Can Google search results show every URL on a domain?

No. Search results can reveal example URLs known to Google, but they are not a guaranteed exhaustive database of the site's URL space. Use them as supplementary evidence, not the main discovery method.

Does a successful crawl mean all pages are indexed?

No. A crawler reports what it discovered and fetched under its settings. Index status must be checked separately, and even a successful recrawl request does not guarantee inclusion in search results.

Should I remove query parameters when deduplicating?

Only when a documented rule establishes that a parameter does not distinguish meaningful content or behavior. Preserve original URLs and record each normalization decision. Some parameters are only for tracking, while others control filters, sorting, or page content.

Is IndexNow a website crawler?

No. IndexNow notifies participating search engines about known URL changes. It does not discover unknown URLs or replace sitemap parsing and internal-link crawling.

What should a website URL crawl export contain?

At minimum, keep the original URL, discovery source, normalized value, response status, redirect destination, and validation time where available. Add canonical and robots findings, content type, crawl depth, index status, and review notes when those fields have been checked.

How do I know when the crawl is complete?

Define the scope and sources before starting. Report unique URLs by source, overlap, errors, exclusions, orphan candidates, and unresolved cases. State that the inventory is complete only relative to the documented boundary and crawl configuration.

Build a Repeatable URL Discovery Process

A reliable URL inventory should remain useful after the first crawl. Save the original records, normalized values, discovery sources, crawl settings, exclusions, and validation timestamps together. This evidence lets you reproduce decisions and distinguish genuine site changes from differences caused by the crawl configuration.

After a migration or major release, compare the new inventory with the last accepted version. Review new URLs, missing records, changed response codes, redirect destinations, and canonical signals. Do not remove a record solely because it disappeared from one crawl. Check whether the source changed, the page became inaccessible, or the crawl rules changed. Keep unresolved differences visible until someone can explain them.

Use the IndexNowFast SEO guides hub for related topics such as crawlability and indexing diagnosis. When the inventory is ready, the separate Google Index Checker can support a point-in-time index-status check for known URLs, subject to its current documented capabilities. Neither resource replaces the discovery and validation steps described above.

The final report should state what was checked, when it was checked, and which gaps remain. A repeatable process provides stronger evidence than a one-time URL count, without claiming to reveal URLs that no available source exposed.