Sitemap Indexes for Large Websites

Use an index when the canonical URL inventory exceeds a single file's 50,000-URL or 50-MB uncompressed limit. An index is also valuable earlier when separate sections have different publishing workflows, owners, or indexation patterns.

Good boundaries match how the site operates: products, categories, articles, locations, videos, or locales. Within a large section, stable ID ranges or hash buckets can keep membership predictable. Numeric chunks are not inherently wrong; repacking every file whenever a page is added makes changes harder to isolate.

Each sitemap element points to a child sitemap. The optional lastmod belongs to the child file itself: it should change when that sitemap's contents change, not whenever the application deploys.

Submit the index, then inspect each child's processing status and discovered-page count. Use the Page indexing report filtered by sitemap for indexing differences. A drop in product-page coverage is a reason to check publication state, templates, canonicals, and rendering—not proof that one of those caused it.

Pair sitemap counts with database counts and response validation. Alert when the difference exceeds a known publishing delay, when a child file stops updating, or when error and redirect rates rise.

Each child allows at most 50,000 page URLs and 52,428,800 uncompressed bytes. The index allows at most 50,000 sitemap locations and the same uncompressed byte limit. Gzip changes neither limit. Google allows up to 500 submitted sitemap index files per site in Search Console.

An index at https://example.com/sitemap_index.xml can reference /sitemaps/products-1.xml on that host. An index under /public/ should reference children in /public/ or deeper, not sibling directories. Files normally belong on the same site; Google documents an exception for properly configured cross-site submission. Do not mix hosts accidentally or expect a checker to infer verified ownership.

Generate and validate children first, make them publicly accessible, then publish the index. Keep a manifest with each child's URL, entry count, uncompressed bytes, and content version. An index pointing to not-yet-deployed files fails even when its own XML is valid.

Check every child after deployment. A 200 index with a missing product shard is not a complete inventory. Compare raw entries and unique page URLs; confirm removals disappeared and additions reached the intended partition. Update index lastmod when the child file changes, not by copying a page timestamp without checking the generated file.

SitemapKit direct extraction has a 50-location ceiling; discovery has a 200-location ceiling and traversal is capped at depth 5. Its 10 MiB transport response limit is below the protocol's 50 MB ceiling. A protocol-valid index can therefore exceed the tool's operational limits. Split audits by known children rather than treating a short export as the whole site.

Keep partial, truncated, stopReason, and the requested budget with each export. Failed children can be skipped without a partial flag, so reconcile against your manifest even on success. Checker validates the supplied index document; it does not recursively certify children. Treat each child as its own validation target.

Avoid generating huge files during the request that serves them. Build or stream from a reliable URL source, cache the result, and refresh it after content changes. A sitemap endpoint that times out under crawler load defeats the discovery mechanism it is supposed to provide.