Google Search Console URL Inspection Tool: Live Tests, Canonicals and Indexing Diagnostics

The Google Search Console URL Inspection tool is one of the most useful diagnostics for understanding how Google has processed a particular URL. It can show whether Google has an indexed record, when it last crawled the page, which canonical URL it selected, and what the current live test can access. Used well, it helps separate a technical access problem from a canonicalization issue, a stale report, or a page that Google has crawled but has not chosen to index.

The most important habit is to distinguish the indexed report from the live test. The indexed report describes information Google recorded during a previous crawl. The live test fetches the current URL and checks certain technical conditions now. Those views can differ because a site has changed, because Google has not recrawled it, or because the live test does not evaluate every indexation question. A successful live test is useful evidence, but it is not a guarantee that Google will index the page.

This guide explains the workflow step by step for WordPress site owners, bloggers, SEO practitioners, and small businesses. It covers access requirements, indexed-state fields, live testing, rendered HTML, page resources, canonical differences, robots directives, structured data, request-indexing limits, common misinterpretations, and a repeatable diagnostic checklist. For wider context, see the My Advisers SEO Services Guide and our technical SEO audit master guide.

What is the URL Inspection tool?

URL Inspection is a Google Search Console feature that reports information about a URL within a verified property. It helps a site owner investigate Google’s stored view of a page and, separately, test whether Google can fetch the current version. The tool is useful when a new page is not appearing in search, an existing page has changed, a canonical URL looks unexpected, or a technical deployment may have introduced a crawl or indexability problem.

URL Inspection is not a general-purpose crawler for arbitrary websites. Access to detailed property data requires the appropriate Search Console permissions. It also does not replace a full site crawl: the tool examines individual URLs, while a crawler can sample or inventory large sets of URLs to find patterns. The strongest workflow combines URL Inspection with sitemap checks, internal-link analysis, server logs, and direct HTTP/HTML testing.

Google’s official URL Inspection tool documentation describes the available information and important limitations. Read it when the interface changes or when a particular status is unclear. Treat the tool as a diagnostic window into Google’s systems, not as a promise that any URL you submit will be included in search results.

Before you inspect a URL

Confirm that the URL belongs to the correct verified Search Console property. A URL can differ by protocol, hostname, subdomain, path, or property scope. For example, a URL under an HTTPS www host is not necessarily represented by a separately verified property for the non-www host. If you inspect the wrong variant, you may be looking at the wrong property or may not have access to the information you expect.

Copy the complete canonical URL from the browser or sitemap rather than typing a shortened path from memory. Keep the query string if it is part of the URL you are investigating, but first decide whether the parameterized URL is expected to be a separate page. When troubleshooting a redirect, inspect both the source URL and the final destination. When investigating a canonical issue, inspect the URL that appears in the report and the canonical target separately.

Record the question you want to answer before starting: Is the page indexed? Can Google fetch it now? Is indexing allowed? Does the current HTML contain the expected content? Does Google select the intended canonical? A focused question helps you avoid treating every warning as an error or requesting indexing repeatedly without fixing the underlying cause.

Indexed state versus live test

The initial URL Inspection result usually represents the information Google has stored from its most recent processing of the URL. It may include the last crawl date, indexed status, user-declared canonical, Google-selected canonical, and information about how Google discovered the page. Because it reflects a previous crawl, it can lag behind the live website. A recent deployment may already have fixed a problem that remains visible in the indexed report.

The live test fetches the current URL and evaluates whether Google can access it and whether certain technical conditions permit indexing. It is useful for validating a fix before waiting for the indexed report to update. However, it does not test every possible reason Google may decline to index a page. It cannot guarantee that the page is not a duplicate, that it will be selected as canonical, or that its content meets all search quality requirements.

Compare the two views instead of choosing one as universally correct. If the indexed state shows noindex but the live test does not, the site may have changed since the last crawl. If the live test fails while the indexed state still says the URL is indexed, the current server response may have regressed. Record the difference, identify when the change occurred, and verify the public response independently.

Understanding common URL Inspection fields

The interface and wording can evolve, but the key questions remain stable: whether Google has an indexed record, whether the page was crawlable, whether the last fetch succeeded, whether indexing was allowed, and how canonicalization was handled. The report may also indicate sitemap discovery or referring pages. Not every field appears for every URL, and some fields are unavailable in the live test because they depend on Google’s stored history.

Field or signal	What it helps you understand	What it does not prove
Indexed status	Whether Google currently reports an indexed record for the URL	That the URL ranks for a particular query or will remain indexed
Last crawl	When Google last fetched the page recorded in the report	That every subsequent change has been processed
Crawl allowed	Whether crawl rules permit the fetch	That the page will be indexed
Page fetch	Whether the relevant fetch succeeded or encountered a response problem	That the content is useful or canonical
Indexing allowed	Whether a detectable directive permits indexing	That Google will choose the URL for the index
User-declared canonical	The canonical URL indicated by the site	That Google will necessarily select it
Google-selected canonical	The URL Google chose as the representative version in its indexed processing	That the site has no other duplication or architecture problems
Sitemap or referring-page information	Some of the paths through which Google discovered the URL	A complete list of every discovery source

Use the fields as evidence, then test the specific hypothesis. A page that is not indexed because it has a noindex directive requires a different fix from a page that Google considers a duplicate of another URL. Do not apply a generic “request indexing” action to every status without understanding what it means.

How to run a live URL test

1. Open Search Console and select the correct verified property.
2. Paste the full URL into the inspection field.
3. Review the stored indexed state before making changes.
4. Choose the live-test option and wait for the fetch and rendering checks to finish.
5. Review crawl availability, indexability signals, and any reported problems.
6. Open the tested page details when available, including HTML, screenshot, resources, and console information.
7. Compare the live result with the indexed result and with a direct request to the public URL.
8. Fix the underlying issue before requesting recrawling, then verify the change again.

A live test is most useful when it answers a defined question. For example, after removing an accidental noindex setting, test the URL to confirm that indexing is allowed. After changing a canonical tag, inspect the returned HTML and compare it with the intended target. After a server migration, test representative URLs from each template and check whether redirects, response codes, and page resources behave as expected.

Do not interpret a passing test as a full SEO audit. The tool is designed for URL-level diagnostics and has documented limits. A page can pass the live test yet remain unindexed because Google has not processed the change, considers it a duplicate, or decides it is not appropriate for search results.

Inspecting rendered HTML

When the tool provides a rendered or tested HTML view, use it to confirm that important content and metadata are actually present in the version Google received. Look for the page title, main content, internal links, canonical element, robots meta tag, and structured data that should be in the response. If a section is missing, compare the output with the source HTML and the browser’s rendered DOM to determine whether the difference comes from JavaScript, delayed content loading, a template error, or a blocked resource.

For JavaScript-heavy websites, inspect both the initial response and the rendered result where the tool exposes them. A human browser may show content after scripts run, while the first response may contain only a shell. Google can render JavaScript, but rendering introduces additional dependencies and potential failure points. Important content and navigation should be implemented so that they remain accessible and discoverable without fragile interaction requirements.

For a deeper explanation of rendering problems, read JavaScript Hydration SEO and the guide to render-blocking resources and LCP. URL Inspection can reveal symptoms, but developers may still need browser dev tools, server logs, and a reproducible test case to isolate the root cause.

Inspecting page resources and console messages

When available, the tested-page details may show resources that were requested while rendering the page. Review failed CSS, JavaScript, images, fonts, and API requests that could affect the main content or navigation. A failed analytics request is usually different from a failed request for the script that renders product descriptions. Prioritize resources by their effect on content, crawlability, and the user’s experience rather than treating every blocked resource as equally serious.

Console messages can reveal uncaught JavaScript exceptions, missing modules, or errors that stop later rendering steps. Reproduce the issue in a browser and inspect the network waterfall to see whether the failure is deterministic, intermittent, or specific to a device or region. If the page relies on third-party scripts, test the fallback behavior when the third party is unavailable.

Do not unblock every resource automatically. Some resources are private or unnecessary, and some failures are harmless. Determine whether Google needs the resource to understand the main content. For a public page that depends on a blocked stylesheet or script, adjust the rules or implementation so the crawler can render it correctly. After deployment, run the live test again and verify the actual rendered output.

Crawl allowed versus indexing allowed

Crawl permission and indexing permission are separate checks. A URL can be crawlable but intentionally noindexed, or it can be blocked from crawling before Google sees the page-level indexing instruction. If a page is crawlable and indexing is allowed, that only means certain technical barriers are absent; it does not force Google to include the URL in search results.

If the live test says indexing is not allowed, inspect the HTML robots meta tag and the X-Robots-Tag response header. Also check whether a plugin, theme, server rule, or CDN is adding the directive. If the page should be indexable, remove the unintended restriction and verify the public response. If the page should be noindexed, confirm that the directive is deliberate and that the sitemap does not continue to advertise the URL as an intended indexable destination.

For details about directives, read Robots Meta Tag for SEO and X-Robots-Tag for SEO. Google’s official documentation on blocking search indexing with noindex explains why a crawler must be able to access a page to see the instruction.

Understanding canonical differences

One of the most valuable uses of URL Inspection is comparing the user-declared canonical with the Google-selected canonical. The user-declared canonical is the site’s preference. Google’s selected canonical is the URL Google chose as the representative version in its indexed processing. A difference does not always mean a bug: the inspected URL may be a duplicate that correctly points to another page. But if Google chooses an unexpected URL, review the signals that help establish the preferred version.

Check the canonical tag, redirect behavior, internal links, sitemap entries, URL parameters, protocol and hostname variants, and the uniqueness of the content. A canonical tag is a signal rather than an unconditional command. If internal links point to one version while the sitemap and canonical point to another, the site is sending mixed signals. Make the preferred URL consistent across all systems and ensure the canonical target is accessible and useful.

For complex cases, use the Advanced Canonicalisation for SEO guide. Inspect both the duplicate URL and the intended canonical URL. Do not automatically redirect every URL with a canonical mismatch; first determine whether the URL should be a duplicate, a permanent move, a distinct page, or an alternate that serves a legitimate purpose.

Discovery information: sitemaps and referring pages

The indexed report may show sitemap information or referring pages that helped Google discover the URL. This is useful when a page appears isolated or when the site owner wants to understand how Google learned about it. However, the information is not guaranteed to be a complete history of every discovery source. Absence of a referring page in the report does not prove that no internal link exists.

If a valuable page is not being discovered, check that it appears in a current sitemap and is linked from relevant indexable pages. A sitemap helps communicate a preferred URL inventory, while internal links provide context and navigable paths. Neither replaces the other. Verify that the sitemap URL is accessible, the listed URL is canonical, and the page is not blocked or noindexed.

See XML Sitemaps for SEO and Orphan Pages in SEO for practical audits. When discovery is the problem, improve the site’s URL architecture rather than relying only on repeated requests through Search Console.

Request indexing: when it helps and what it cannot do

After publishing a new page or correcting a technical issue, you may use the Request Indexing action when it is available and appropriate. It can notify Google that a URL is ready to be recrawled, but it does not guarantee that Google will crawl the page immediately or include it in the index. Requests are subject to limits and prioritization, and repeated submissions of the same unchanged URL are not a reliable way to accelerate processing.

Before requesting indexing, make sure the page returns the expected response, is crawlable, has no unintended noindex directive, uses the intended canonical, and contains useful content. If the URL redirects, request inspection of the final destination. If the issue affects thousands of URLs, fix the template or sitemap rather than manually submitting every URL one at a time.

For a newly published article, a practical sequence is to link it from a relevant category or existing article, ensure it appears in the sitemap, inspect it once, and request indexing if the test indicates that it is eligible. Then monitor the indexed state and performance data over time. Avoid promising same-day indexing, because processing time depends on Google’s systems and the site’s circumstances.

Why a successful live test may still not lead to indexing

A live test checks technical access and certain indexability signals. It does not guarantee that Google will select the page for its index. Google may treat the URL as a duplicate, choose another canonical, consider the content too similar to existing pages, or decide that the page does not merit inclusion for other reasons. The tool may also be unable to evaluate every issue that affects the final indexing decision.

If a page passes the live test but remains unindexed, move beyond the test result. Compare the page with the canonical target, inspect content overlap, strengthen contextual internal links, confirm that the URL is in the correct sitemap, and review whether it satisfies a clear user need. Look at groups of similar URLs to see whether the issue is sitewide or limited to one page. Avoid making repeated technical changes when the evidence points to a content or duplication problem.

Remember that crawling, rendering, indexing, and ranking are separate stages. A page can be fetched without being rendered fully, rendered without being indexed, or indexed without receiving meaningful impressions. Each stage has different diagnostics and remedies. A clear workflow prevents the team from confusing a successful fetch with a successful search outcome.

How to diagnose a noindex problem

When URL Inspection reports that indexing is not allowed, first verify whether the directive is intentional. Inspect the live HTML for <meta name="robots" content="noindex"> or a crawler-specific variant, and inspect the HTTP response for an X-Robots-Tag header. Check the SEO plugin, page settings, theme template, custom code, CDN, and server configuration to find the source.

If the page should be indexed, remove the unintended rule, clear caches, fetch the URL again, and run a new live test. Ensure robots.txt does not block the URL from being crawled, because a crawler must be able to read the updated directive. Check whether the sitemap still includes the URL and whether canonicalization points to the right destination. Once the public response is correct, request recrawling if appropriate and monitor the indexed state.

If the page should not be indexed, document the reason and make sure the sitemap and internal architecture reflect that decision. Noindex is not access control; if the page contains confidential material, protect it with authentication or remove it from public access. The technical configuration should match the business requirement, not merely satisfy a status label.

How to diagnose a robots.txt block

If crawling is blocked, inspect the robots.txt file for the exact host and protocol being tested. A rule on one hostname does not necessarily apply to another. Check the relevant user-agent group and the path pattern, including case sensitivity and wildcard behavior. Do not assume that a broad Disallow rule applies only to the URL you intended.

Use the current Google robots.txt specification guidance to understand matching behavior. If the URL should be crawled, adjust the rule carefully and verify the resulting file. A robots.txt change can affect many URLs, so test the intended URL and a small set of neighboring paths before deployment.

For a page that needs to be removed from search, do not use a robots.txt block as the only method if the crawler must read a noindex directive. The correct sequence may be to allow crawling, return noindex, and wait for the search engine to process the change. Crawl management and indexation management should be treated as distinct tasks.

How to diagnose server errors and redirects

If the fetch fails, inspect the HTTP response independently. A 5xx response may indicate origin-server instability, application errors, rate limiting, or an intermediary problem. A redirect may be expected, but a chain, loop, or redirect to an unrelated page can prevent the intended URL from being processed correctly. A 404 or 410 may be appropriate for removed content, but it should be intentional.

Check server logs and CDN logs for the request time, status code, user agent, and response latency. Compare the result from a normal browser with the result from the public server and any relevant inspection tool. Some systems return different responses based on cookies, geography, authentication, or bot filtering. Do not conclude that a page is healthy simply because it loads for an administrator who is logged in.

Use the Soft 404, 500 Errors and Redirects guide to choose the right response for the URL lifecycle. After fixing the server or redirect, run the live test again and monitor whether Google recrawls the URL. If many URLs fail together, investigate the shared infrastructure rather than editing individual pages.

Structured data and enhancement diagnostics

Depending on the URL and the features Google detects, Search Console may show structured-data or enhancement information. Use those reports to identify parsing errors or missing required properties for eligible search features. The presence of valid structured data does not guarantee a rich result, and URL Inspection is not a substitute for checking the relevant structured-data report or testing tool.

When a page’s markup changes, verify the JSON-LD or other supported structured data in the served HTML. Check that values match visible page content, that the markup describes the correct entity, and that the page is accessible to the crawler. Avoid adding schema simply to obtain a visual enhancement when the page does not meet the feature’s eligibility requirements.

If the page uses client-side rendering, compare the markup before and after rendering. Ensure that important structured data is not duplicated or changed unpredictably by scripts. When errors appear, fix the underlying template or data source and retest a representative set of pages rather than manually patching every URL.

Diagnosing content that is missing from Google’s rendered page

If the rendered HTML or screenshot lacks important content, determine whether the content is absent from the initial response, loaded through JavaScript, hidden behind user interaction, or blocked by an inaccessible resource. Search for the expected text in the HTML, review the page resources, and inspect the console for exceptions. A visible browser page may rely on cookies, local state, or third-party APIs that the inspection environment does not reproduce.

For critical content such as product descriptions, service details, navigation links, and article text, prefer a robust rendering approach. Server-side rendering or static rendering can reduce dependence on client-side execution, although the correct choice depends on the application. If client-side rendering is necessary, make sure the page can render consistently and that links are exposed as standard crawlable anchors.

Do not assume every difference between the browser and the inspection tool is an SEO defect. Some differences arise from personalization or unavailable user state. Focus on whether the crawler can access the primary content and links required to understand the page. Test across the key templates and devices that matter to the site.

Using URL Inspection after a WordPress publication

For a new WordPress post, first verify that the post is published rather than a draft, that its permalink is stable, and that the page returns a successful public response. Confirm that the SEO plugin or theme has not added noindex, that the canonical URL matches the permalink, and that the post appears in the appropriate sitemap. Then add a contextual internal link from a relevant category page or existing article.

Inspect the page in Search Console after the technical basics are correct. Review the stored state, run a live test if needed, and check the rendered HTML and resources. If the page is eligible but not indexed, do not assume that the tool has failed. Google may need time to process it, or the page may need stronger unique value and internal discovery signals.

For related WordPress technical guidance, read XML Sitemaps for SEO, Robots Meta Tag for SEO, and Critical CSS for WordPress. Together, these checks cover several common causes of crawl and rendering problems.

How to investigate many URLs efficiently

URL Inspection is best for investigating representative URLs and high-priority cases, not for manually auditing every URL on a large site. Start with a crawler export, sitemap list, Search Console reports, analytics landing pages, and server logs. Group URLs by template and issue type, then select samples that represent the pattern. Use URL Inspection to investigate the most important examples and confirm whether the technical hypothesis is correct.

If a shared template emits noindex, inspecting one representative URL can reveal the source, but you should still sample different templates and content types to confirm the scope. If a canonical mismatch affects a parameter pattern, test several parameter combinations. If server errors affect one directory, compare URLs across the directory and inspect logs around the failure period.

Build an issue register with columns for URL, template, observed status, expected status, robots state, canonical target, sitemap membership, last crawl, business priority, root cause, owner, fix, and verification date. This turns URL Inspection results into actionable work rather than a collection of screenshots without a clear outcome.

Common mistakes when using URL Inspection

• Confusing indexed state with live state. The stored report may reflect a previous crawl; the live test checks the current response.
• Assuming a passing live test guarantees indexing. It confirms only certain technical conditions.
• Requesting indexing before fixing the issue. A request cannot repair a noindex tag, broken redirect, or server error.
• Repeatedly submitting the same URL. Repetition does not guarantee faster processing.
• Inspecting the wrong URL variant. Protocol, host, path, and parameters can change the URL being tested.
• Ignoring canonical differences. A Google-selected canonical may indicate duplication or conflicting signals.
• Checking only the homepage. Different templates can produce different technical outcomes.
• Trusting the editor preview. The public response and crawler view are what matter.
• Assuming all errors are technical. Some pages are not indexed because they are duplicates or provide insufficient distinct value.
• Using a single tool in isolation. Combine URL Inspection with sitemaps, internal links, logs, and direct response checks.

Most of these mistakes come from treating URL Inspection as a button that makes pages appear in search. It is more useful as a diagnostic workflow: inspect the evidence, form a hypothesis, verify the public response, correct the underlying cause, and monitor the outcome.

A practical diagnostic decision tree

1. Is the URL in the correct property? If not, select the right verified property and inspect the exact canonical URL.
2. Is the URL already indexed? If yes, compare the selected canonical and last crawl with the current site state. If no, continue investigating the reason.
3. Can Google crawl the URL? If no, inspect robots.txt, authentication, firewall rules, and server responses.
4. Did the fetch succeed? If no, investigate redirects, DNS, timeouts, 4xx/5xx responses, and intermittent failures.
5. Is indexing allowed? If no and the page should be indexed, locate the meta or header directive and remove the unintended rule.
6. Does the live HTML contain the main content? If no, investigate rendering, scripts, resources, and template output.
7. Does the canonical match the intended URL? If not, review canonical tags, redirects, internal links, and sitemap entries.
8. Does the page provide distinct value? If technical checks pass but indexing does not occur, review duplication, content quality, and discovery signals.
9. Have you fixed and retested? If yes, request recrawling where appropriate and monitor the indexed report rather than resubmitting repeatedly.

This sequence prevents premature conclusions. It starts with access and response integrity, then checks indexability and canonicalization, and only then moves toward content and selection questions. That order usually makes troubleshooting faster because it eliminates foundational failures before investigating more subtle causes.

How to document and report findings

A useful technical report states what was observed, where the evidence came from, what the expected behavior should be, and what change is recommended. Avoid writing “Google does not like this page” when the evidence only shows that a noindex directive is present. Be precise: “The live HTML currently includes noindex, while the business requirement is to keep the page eligible for indexing.” This wording leads directly to a testable fix.

Include the inspection date, exact URL, property, indexed state, last crawl if shown, live-test result, declared and selected canonicals, response status, robots directives, sitemap membership, and relevant resource errors. Add a screenshot or export only when it helps reproduce the finding. Keep sensitive property data within the authorized team.

After implementation, record the changed setting or code, the verification result, and the date of the next follow-up. A successful live test can confirm that a technical fix is now visible to the tool, while the indexed report may take longer to reflect it. Reporting these separately avoids overpromising and helps stakeholders understand the difference between a fix being deployed and a search outcome being processed.

Frequently asked questions about URL Inspection

Does URL Inspection show the exact page Google has indexed?

The indexed view reflects Google’s stored information from a previous crawl, including available crawl and canonical details. It is not a guarantee that every field represents the current live page. Use the live test to check the current response, and compare both views when the website has changed.

Does a successful live test mean the URL will be indexed?

No. The live test checks technical access and certain indexability conditions, but it does not guarantee that Google will select the page for indexing. Duplicate content, canonical selection, content value, and other processing factors can still affect the outcome.

Should I request indexing every time I update a page?

Not necessarily. Use the action when a meaningful change or new URL merits a recrawl request, but do not rely on repeated submissions for routine edits. Ensure the sitemap and internal links are accurate and allow Google’s normal discovery systems to operate.

Why does Google show a different canonical URL?

Google may consider another URL a better representative of duplicate or similar content. Review canonical tags, redirects, internal links, sitemap entries, and parameter variants. Make the preferred URL consistent across those signals and confirm that the target is accessible.

Why does the live test pass while the indexed report still shows an error?

The indexed report can describe a previous crawl, while the live test evaluates the current response. If you fixed the problem recently, the stored report may not have updated yet. Confirm the current HTML and headers, then monitor recrawling and indexation.

Can URL Inspection test a competitor’s website?

The detailed Search Console URL Inspection report requires access to the relevant verified property. It is not designed to provide private indexing data for any arbitrary competitor URL. Public URLs can still be examined with ordinary browser and web tools, but those checks do not expose Google’s property-specific report.

Does the tool replace a site crawler?

No. URL Inspection is valuable for individual URL diagnostics, while a crawler inventories many pages and identifies patterns. Use a crawler to find issues at scale, then use URL Inspection to investigate representative URLs and confirm Google’s current view.

Can it identify every reason a page is not indexed?

No. It exposes useful technical and indexing information, but it does not reveal every internal decision or guarantee a complete explanation for all cases. Combine it with content review, internal-link analysis, sitemap audits, server logs, and other Search Console reports.

How long does Google take to recrawl a URL after a request?

There is no guaranteed fixed time. Processing depends on Google’s systems and the site’s circumstances. A request may help communicate that a URL is ready, but it does not guarantee immediate crawling or indexing.

What should I check first when a published WordPress post is not indexed?

Confirm that it is publicly published, returns a successful response, has no unintended noindex directive, uses the intended canonical, appears in the sitemap, and is linked from relevant pages. Then run URL Inspection, compare the indexed and live states, and review the content’s distinct value.

Authoritative references and further reading

• Google Search Console Help: URL Inspection tool
• Google Search Central: Debugging your pages
• Google Search Central: Discover Google-selected canonical URLs
• Google Search Central: Block Search indexing with noindex
• Google Search Central: Robots meta tag specifications
• Google Search Central: Build and submit a sitemap
• Bing Webmaster Tools: URL Inspection
• Google Search Central: Introduction to robots.txt

Final recommendations

Use URL Inspection to answer a focused diagnostic question, not as a substitute for a complete SEO strategy. Compare the indexed state with the live test, inspect the rendered HTML and resources when relevant, and verify the canonical, robots directives, sitemap inclusion, and HTTP response. Fix the underlying cause before requesting recrawling, and remember that a successful test does not guarantee indexation.

For WordPress, make sure the page is published, its permalink is stable, the public response matches the intended configuration, and the URL is discoverable through a sitemap and relevant internal links. For large sites, use crawlers and logs to find patterns, then use URL Inspection to investigate the highest-priority examples. Record findings and retest after deployment so the team can distinguish a completed technical fix from a search engine’s later processing decision.

My Advisers provides practical technology consulting and digital growth guidance for startups, professionals, bloggers, and small and medium-sized businesses. Continue with our guides to XML sitemaps, advanced canonicalisation, robots meta tags, and crawl demand versus crawl budget to build a consistent crawling and indexing workflow.