Crawlability Test: How to Check a URL and Diagnose Failures

Quick Answer

A crawlability test checks whether a search engine crawler can reach a URL, follow redirects, access the page under robots.txt rules, and read its content. A complete test also checks indexing rules, canonical URLs, and rendered content. Passing these checks does not prove that Google or Bing has indexed the page.

Key Takeaways

A good crawlability test checks several parts of a page. Each result answers a different question about access, content, or search visibility.

• An HTTP 200 response shows that a request succeeded, but it does not prove the page is indexed.
• Redirect tests must follow every redirect and check the final URL, status, and content.
• Robots.txt rules can block a specific search crawler even when the page opens in a normal browser.
• A noindex directive controls indexing eligibility, while robots.txt mainly controls crawler access.
• Canonical tags and JavaScript rendering can reveal problems that a basic HTTP test misses.
• Google Search Console and Bing Webmaster Tools provide different records for their own search systems.
• After fixing a technical issue, run the test again and check stored indexing results separately.

The goal is to find the failing part of the process. Fix that problem before requesting another crawl or submitting the URL again.

Run a Crawlability Test in 10 Steps

Start with one exact URL and test each part in order. First, confirm that a public client can reach the page. Next, check crawler rules, indexing signals, content, and discovery. Finish with search-engine reports when you have site access.

This order saves time. A page with a server error needs a different fix from one that has a noindex tag. Record each result instead of giving the whole page a single pass or fail score. The following workflow is a practical method based on official guidance, not an official Google or Bing certification test.

Step 1: Record the URL and Expected Canonical

Write down the full URL you want to test. Include its protocol, hostname, path, and any query string that matters.

Also record the intended canonical URL. This is the main page address that you want search engines to treat as the preferred version.

For a repeatable test, record the date, time zone, test tool, and user agent. The user agent identifies the client making the request.

A URL starting with HTTP is not the same address as its HTTPS version. The same applies to www and non-www hosts. Redirects may connect these versions, but you still need to check where they lead.

Expected result: The requested URL and intended final page are clear before testing begins.

Step 2: Test Public Access and Redirects

Open the URL using a client that is not signed in. Check whether the server responds without a password, browser session, or private network connection.

An HTTP inspection tool can show the response and redirect chain. For example, a terminal command such as curl -IL https://example.com/sample-page can help inspect headers and redirects. A HEAD request may behave differently from GET, so use a normal GET request when you need to confirm the page content.

Record every redirect hop, its status, and the final URL. A 301 redirect may be correct when moving from HTTP to HTTPS. It is a problem if it leads to the wrong page or enters a loop.

Expected result: The URL reaches the intended public destination without an unexpected redirect or access failure.

Step 3: Read the Final HTTP Response

Check the final status code and the content returned. Google's HTTP status code guidance explains how different responses affect crawling.

A 2xx response normally means the server handled the request successfully. A 404 means the requested resource was not found. A 429 signals too many requests or rate limiting. A 503 commonly signals temporary server unavailability.

Google may slow crawling when it encounters 429 or 5xx responses. Long-lasting failures can also affect pages already in its index.

Do not treat every 404 as a defect. A removed page may be meant to return that status. What matters is whether the response matches the page's intended state.

Expected result: The final response is stable and appropriate, and the returned content matches the requested resource.

Step 4: Check robots.txt for the Intended Crawler

Robots.txt tells compliant crawlers which paths they may request. The IETF Robots Exclusion Protocol, RFC 9309, defines the standard format and rule matching behavior.

The file normally lives at the root of the site, such as https://example.com/robots.txt.

For a site-specific starting point, you can inspect the IndexNowFast robots.txt URL using an HTTP client. This is an example of the path to test, not a claim that its current response has been checked. Record the actual result before drawing any conclusion.

Check the file response, the applicable user-agent group, and the rule that matches your URL. Rules may differ for Googlebot, Bingbot, and other crawlers.

For instance, Disallow: /private/ in the applicable group can prevent a compliant crawler from fetching paths under that folder.

A missing robots.txt file is not the same as a server failure while retrieving the file. RFC 9309 treats those cases differently. Search engines also publish details about their own handling.

Google's robots.txt documentation explains another limit: blocking crawling does not reliably remove a URL from search results.

Expected result: The intended crawler is allowed to fetch the page, unless blocking is deliberate.

Step 5: Inspect noindex and Response Headers

A page may return HTTP 200 and still tell search engines not to index it.

Check the page's HTML for a robots meta tag. Also inspect HTTP response headers for X-Robots-Tag. Either location can carry an indexing instruction.

For example, <meta name="robots" content="noindex"> asks supporting search engines not to index that page.

A noindex instruction is not the same as a robots.txt block. Google generally needs to crawl the page or receive the relevant response header to see the directive.

If robots.txt blocks access, Google may never see a page-level noindex tag. That can leave a discovered URL eligible to appear without its page content.

Expected result: The page has no unintended noindex instruction when indexing is the goal.

Step 6: Compare Canonical Signals

A canonical URL tells search engines which version of similar content the site prefers. It is a signal, not an absolute command.

Compare the page's rel="canonical" value with its final redirect destination, internal links, and sitemap entry.

Imagine that a product page redirects to a new address, but its canonical tag points to the old address. The signals now disagree. A search engine may choose a different page version from the one you expected.

Check whether the canonical target exists and serves the intended content. Make sure accidental test domains, old paths, and staging addresses have not been left in production markup.

Expected result: The site's main signals agree on the preferred page address.

Step 7: Check Rendered Main Content

A successful fetch does not prove that a crawler can see the main text.

JavaScript-powered websites may send a nearly empty HTML page first. The browser then runs scripts to show content. Google supports JavaScript rendering, but a script failure or blocked resource can still affect what it processes.

Compare the initial HTML with the rendered page. Look for the main heading, body text, internal links, title, and important metadata.

Check the Google JavaScript SEO guidance when essential content appears only after scripts run.

A rendering tool may use a different browser or user agent from Googlebot. Record which tool produced the result. Do not claim a generic browser test proves exactly what Google saw.

Expected result: The tested rendering exposes the page's intended content and important links.

Step 8: Check Sitemap and Internal Discovery

A search engine needs a way to learn that a URL exists. Internal links and XML sitemaps can help.

Find at least one sensible internal path to an important public page. A link from a relevant category or service page is often more useful than leaving the URL isolated.

Then check whether the preferred URL appears in the XML sitemap. Google's sitemap building guidance calls for fully qualified URLs and explains that submitting a sitemap does not guarantee crawling or indexing.

A sitemap file has a limit of 50,000 URLs or 50 MB when uncompressed. Larger sites can use multiple files and a sitemap index.

Remove unintended redirects, errors, and excluded URLs from the sitemap when maintaining a list of pages meant for search.

Expected result: The page has suitable discovery paths and its sitemap entry agrees with the intended canonical.

Step 9: Compare Google and Bing Evidence

Use search-engine-owned reports to answer questions that public crawlers cannot answer.

For a verified property, Google Search Console URL Inspection can show Google's stored information about a URL. Its live test checks the current page under specific conditions.

These results can differ. A recent fix may pass a live test while Google's stored record still reflects an older crawl. A positive live test is not a guarantee of indexing.

For Bing, Bing Site Explorer provides URL-level information based on Bing's own data.

Keep the two engines separate. A Google result cannot prove what Bing has indexed. Likewise, a Bing crawl result is not Google's decision.

Expected result: Record each engine's actual report, including its date and any stated limits.

Step 10: Fix, Retest, and Save the Results

Return to the first failed check after making a change. Repeat it using the same tool and settings where possible.

Compare the original observation with the new result. Record what changed, when the fix went live, and which test now passes.

Do not mark an untested condition as successful. Use Not tested when the tool, access, or evidence was unavailable.

Save the final report so another team member can repeat the check. Review the search engine's stored information later if indexing status remains important.

Expected result: The original defect no longer appears in a repeat test, or the report clearly explains what still needs work.

Crawlability Test Results: A Completed Failure Matrix

A crawlability test can return several different failures. Each points to a different cause and next action. An HTTP 200 result is useful, but it cannot replace every other check.

The matrix below uses hypothetical technical examples based on documented crawler behavior. These are not real customer tests or measured failure rates.

The rows separate observed facts from what remains unknown. A 404 may be correct for deleted content. A noindex tag may be correct for private search pages or other content not meant for search.

Classify each issue as access, crawl policy, indexing policy, content, discovery, or stored search-engine state. This makes it easier to assign the right fix. If a test cannot run, record Not tested instead of guessing.

Choose the Right Tool for the Test

No single tool answers every crawlability question. Start with the evidence you need, then choose the tool that can provide it.

A basic HTTP client can inspect a public URL. An SEO crawler can test many pages. Search Console and Bing Webmaster Tools add information held by the search engines themselves.

Microsoft's Bing Site Scan documentation describes scan scope, crawl depth, speed, and issue reporting.

For one missing product page, a public response check plus an owner-only URL inspection may be enough to start. A site migration involving thousands of changed URLs needs broader checking and a clear record of failures.

Use a third-party crawler to find technical symptoms. Use the engine's own reports when you need evidence about its observed crawl or index state. Neither type of tool replaces the other.

Fix the Failed Layer Before Requesting Another Crawl

Resubmitting a URL does not fix broken access, blocked crawling, missing content, or wrong canonical signals. The useful next step depends on what the test found.

The IndexNowFast guide to pages Google is not indexing covers broader indexing problems. For a crawlability test, focus first on the exact failure shown by the evidence.

Access and Server Failures

Fix access problems before changing SEO settings. If a public request fails, check the server, network, authentication, firewall, and content delivery network settings.

A firewall may allow ordinary visitors but challenge certain automated clients. Rate limits can also affect crawlers during heavy traffic or repeated requests.

Check server logs when available. They can help show which client made a request and how the server handled it. A generic HTTP test cannot prove that Googlebot itself made a request.

For redirects, correct the final destination and remove unwanted loops. Update internal links to point to the preferred address. A broken destination should not be treated as healthy just because the first URL returns a redirect.

Robots.txt and noindex Conflicts

Decide whether the page should be crawled, indexed, both, or neither.

If you want Google to crawl and consider indexing a public page, remove unintended crawl blocks and exclusion directives. Confirm that the correct user-agent rules apply.

If the page should stay out of Google's index, follow its documented noindex or access-control methods. Do not assume robots.txt alone will keep the URL out of search.

An important trap occurs when a page has noindex and robots.txt blocks Googlebot. The crawler may be unable to read the exclusion instruction. Fix the policy based on the actual goal, rather than removing both rules without review.

Canonical, Rendering, and Discovery Problems

A page can pass its access test and still have the wrong preferred address. Compare the canonical tag, redirects, internal links, and sitemap.

If essential text appears only after JavaScript runs, check the rendered output and script errors. A visible page in your browser may depend on saved sessions or resources that another client cannot use.

For a page with no useful internal links, add a relevant link from an established page. Keep its sitemap entry accurate.

Once the technical issue is resolved, the IndexNowFast URL submission guide explains when submitting a URL may help with discovery. Submission is not a substitute for fixing the page.

Confirm the Fix Without Mistaking a Live Pass for Indexing

A successful retest proves that a specific technical condition changed. It does not prove that a search engine has updated its index.

Google's URL Inspection troubleshooting guidance helps explain why live results and stored records may differ.

There are three useful outcomes to record. The original issue may still exist. It may be fixed while the stored index record remains old. Or the engine may report a different indexing issue that needs separate review.

Worked Example: A Page With an Accidental noindex Tag

Consider a made-up page at https://example.com/sample-page. Assume the owner wants it included in search, but its HTML contains noindex.

This completed example shows how a result record should read. All observations below are assumed for teaching. No live test was performed.

The example shows why the report needs separate fields. HTTP access can pass while indexing eligibility fails.

A real report should also include the test timestamp, time zone, user agent, evidence location, and deployment time. Save both the original and later result. Never replace an untested engine observation with an assumed success.

For bulk follow-up, the IndexNowFast Google Index Checker describes point-in-time index checks and CSV export. Its public page describes checking up to 10,000 URLs. These are vendor-described features, not proof of live crawler access or indexing outcomes. The checker should not be confused with the full technical test in this guide.

Crawlability Test FAQs

A crawlability test often raises questions about indexing, crawler access, and how to judge a result. The answers below explain what common results do and do not establish. They also show which tool or action fits each situation. Start with the actual test result rather than assuming that a missing search listing means the crawler failed. Public tests, live inspections, and stored search-engine reports each have different limits. Keep those limits clear when reporting findings to a website owner.

Does HTTP 200 mean Google can index the page?

No. HTTP 200 shows that the tested request succeeded. Google still needs appropriate crawl access and page content. Noindex instructions, canonical selection, and other indexing conditions can affect inclusion. A successful response is not an indexing guarantee.

Can a URL pass a crawlability test but remain unindexed?

Yes. A page may be reachable, crawlable, and free from obvious technical blocks without appearing in search. Search engines make separate indexing decisions. Check their own reports before deciding what needs attention.

Can I run a crawlability test without Search Console?

Yes. You can inspect public HTTP responses, redirects, accessible robots.txt rules, HTML directives, and some rendered content. Google-specific stored index information requires access to the relevant Search Console property. Server logs and other private data also need suitable access.

Does submitting a sitemap make a page crawlable?

No. A sitemap helps search engines discover URLs. It cannot fix a server error, blocked path, or missing content. Sitemap submission does not guarantee that every listed page will be fetched or indexed.

Can robots.txt block Google but allow Bing?

Yes. Robots.txt can define different rules for different user agents. Test the rule that applies to each intended crawler. A general browser request does not prove how either search engine handled the page.

Can Google crawl JavaScript pages?

Yes. Google supports JavaScript rendering. Still, scripts may fail or required resources may be unavailable. Check whether the main content appears in the relevant rendered output. Do not rely only on what a signed-in browser displays.

Should I resubmit a URL after every crawlability failure?

No. First fix the problem. A new submission cannot repair a blocked crawler, failed server request, or incorrect page directive. Once the page is ready, use the appropriate discovery or submission method if needed.

Is a passing crawlability test enough for SEO success?

No. A passing test helps establish technical access and basic eligibility. It does not promise indexing, rankings, or traffic. Content value, search-engine decisions, and other factors remain separate from crawlability.

Make Crawlability Testing Repeatable

A useful crawlability test produces evidence, not just a green checkmark. It should show the exact URL, tested crawler, response, directives, content state, and any observed failure.

Keep the results from before and after each fix. That record helps developers, site owners, and SEO teams discuss the same facts. It also prevents a successful HTTP request from being mistaken for confirmed indexing.

If several pages fail the same test, group them by the shared cause. Fixing one server rule or template error may be more useful than submitting every URL again.

A repeatable report should make untested checks visible. For example, a page may pass a public HTTP test while its Google index state remains unknown. Record that difference rather than giving the URL a full pass. When the owner later gets Search Console access, the stored result can be added without changing the earlier evidence.

Technical readiness comes first. Discovery and search-engine indexing remain separate steps. Use the relevant search engine's own reports to confirm what happened after the fix, without assuming a guaranteed result or timeline.

SEO Title: Crawlability Test: How to Check URLs and Fix Issues

Meta Description: Run a crawlability test with 10 clear steps. Check HTTP errors, robots.txt, noindex, rendering, and Google or Bing reports to find the right fix.