Sitemap Indexes for Large Websites
How to organize millions of URLs into stable child sitemaps that are easier to generate, crawl, and diagnose.
Key takeaways
- Split sitemaps by stable operational boundaries, not arbitrary daily batches.
- Keep child files below protocol limits with predictable headroom.
- Monitor submitted and indexed counts per child sitemap.
- Do not nest indexes recursively for convenience; keep discovery simple.
When a sitemap index becomes necessary
Use an index when the canonical URL inventory exceeds a single file's 50,000-URL or 50-MB uncompressed limit. An index is also valuable earlier when separate sections have different publishing workflows, owners, or indexation patterns.
Choose stable partitions
Good boundaries match how the site operates: products, categories, editorial articles, locations, videos, or locales. Stable partitions make a coverage drop actionable. Arbitrary chunks such as sitemap-17.xml tell you only that a number changed.
- Keep the same URL in one child sitemap at a time
- Leave headroom for growth instead of filling every file to the limit
- Use predictable names that describe the content type
- Regenerate only the child files whose inventory changed
A valid index document
Each sitemap element points to a child sitemap. The optional lastmod belongs to the child file itself: it should change when that sitemap's contents change, not whenever the application deploys.
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/sitemaps/products-1.xml</loc>
<lastmod>2026-07-18</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemaps/guides.xml</loc>
<lastmod>2026-07-16</lastmod>
</sitemap>
</sitemapindex>Monitor sections independently
Submit the index, then inspect child sitemap coverage. A products sitemap with a sudden fall in indexed URLs points to a product-specific template, canonical, inventory, or rendering issue. This is much faster to diagnose than a global coverage change.
Pair sitemap counts with database counts and response validation. Alert when the difference exceeds a known publishing delay, when a child file stops updating, or when error and redirect rates rise.
Large-site failure modes
Avoid generating huge files during the request that serves them. Build or stream from a reliable URL source, cache the result, and refresh it after content changes. A sitemap endpoint that times out under crawler load defeats the discovery mechanism it is supposed to provide.
Map child sitemap files and inspect their URL counts.
Open the free toolFrequently asked questions
How many URLs can one sitemap contain?
Up to 50,000 URLs and 50 MB uncompressed. Use multiple files before either limit is reached.
How should large sitemaps be split?
Prefer stable content types, locales, or application partitions that help you diagnose coverage and regenerate only changed sections.
What does lastmod mean in a sitemap index?
It describes the last meaningful modification of the referenced child sitemap file, not the deployment time of the website.
Primary references
See our technical methodology and editorial policy.