Indexability 8 min read Updated August 2026

Indexability: robots meta, X-Robots-Tag, and crawl access

A URL can be crawlable but not indexable, blocked from crawling before its noindex directive is seen, or technically indexable while canonicalized elsewhere. The report evaluates robots meta tags and X-Robots-Tag together, including crawler-specific values.

At a glance

Crawling, rendering, indexing, and serving are different decisions

Use robots.txt to manage crawling and page-level noindex to exclude a fetched resource from search. Then verify status, canonical, content, and authentication because the absence of noindex alone does not guarantee indexing.

  • Do not block a URL in robots.txt when a crawler must see its noindex.
  • Combine all applicable robots directives and obey the most restrictive result.
  • Check both HTML meta tags and HTTP X-Robots-Tag headers.
  • Treat “indexable” as a technical prerequisite, not an indexing guarantee.
01

The controls have different jobs

  • robots.txt Controls crawler access to URL paths; it is not a reliable instruction to remove a URL from search.
  • robots meta Applies indexing and preview directives to HTML when the page can be fetched.
  • X-Robots-Tag Applies equivalent directives through HTTP and works for HTML and non-HTML resources.
  • Canonical Signals which equivalent URL should represent a cluster; it is not a substitute for noindex.
02

Resolve directives by crawler and location

A page may contain a general robots rule, a Googlebot-specific rule, and one or more HTTP header values. Normalize directive names, identify the user agent each value targets, and combine them instead of stopping at the first tag.

Unexpected headers often come from a CDN, staging middleware, security plugin, or framework response layer. Inspect the final public response after redirects rather than only the page template.

03

Investigate why an allowed page is absent

After removing noindex, confirm that the URL returns 200, is not login-protected, is not canonicalized to another URL, has useful visible content, and is discoverable through links or a sitemap.

Use URL Inspection for Google’s fetched response, rendered page, selected canonical, and indexing status. Allow time for recrawling; changing a tag does not update the index immediately.

Action checklist

Audit indexability as a complete decision path

  1. 01Check robots.txt access for the exact URL and required resources.
  2. 02Inspect every robots meta and X-Robots-Tag value.
  3. 03Confirm final status, authentication, and canonical target.
  4. 04Make the URL discoverable through crawlable links or a sitemap.
  5. 05Validate the live result with URL Inspection after recrawling.

Official references

The technical recommendations in this guide are aligned with these primary sources.