How to Find a Website’s Sitemap

Open the site's /robots.txt file and look for one or more Sitemap directives. The value is normally an absolute URL and can point to either a sitemap or a sitemap index. This is the most reliable public hint because site owners can declare non-standard locations there.

If robots.txt contains no declaration, test a small set of conventional paths. Do not assume a 200 response is valid: some sites return an HTML error page with status 200, so inspect the content type and verify that the document contains a urlset or sitemapindex root.

Do not trust the filename or status code alone. A missing sitemap can return a branded HTML page with status 200, while a real sitemap may redirect to a CDN or use application/xml, text/xml, or a gzip content type. Record the final URL and headers, then inspect the downloaded body for a urlset or sitemapindex root.

Keep headers and body separate, then parse the XML rather than searching for a tag with a regular expression. The root check below is only a first pass; namespace, entries, metadata, and limits need further validation. For a .gz download, decompress the file before running xmllint. curl --compressed handles HTTP content encoding, not every gzip file served as a download.

WordPress commonly exposes /wp-sitemap.xml or a plugin-generated /sitemap_index.xml. Shopify stores a sitemap at /sitemap.xml. Other platforms may generate files dynamically or expose a custom index. Platform detection narrows the search, but robots.txt and content validation remain the source of truth.

Search Console's Sitemaps report shows submitted files and processing status. Your CMS, deployment pipeline, or SEO plugin may also expose the generated location. This is preferable to guessing because private staging rules, reverse proxies, and subfolder installations can change the public path.

A responsible finder checks robots.txt, follows declared sitemap URLs, probes a bounded list of common paths, and stops when it has enough evidence. It should deduplicate normalized URLs, follow sitemap indexes recursively with limits, reject non-HTTP targets, and use short timeouts.

SitemapKit discovery uses the supplied URL's origin: https://example.com/shop does not limit it to /shop. It probes robots.txt and a fixed list of 16 common paths, not every possible filename. Robots declarations and index children are candidates; common-path responses must contain a sitemap-root marker. A returned location is not proof that the file meets every protocol rule.

The discovery coordinator caps its inventory at 200 locations and index traversal at depth 5, within a default 45-second foreground budget. Direct extraction has a separate 50-location ceiling. These are tool limits, not the protocol's 50,000-entry limit. Inspect partial, truncated, and stopReason when present, then check large indexes in smaller groups.

Discovery deduplicates www and apex sitemap identities and repeated path separators. If those hosts serve different inventories, check the exact files independently. A robots declaration can remain in the result even if its fetch later fails. An empty result or partial false is not proof that no sitemap exists or that every candidate loaded.

Test the preferred HTTPS hostname and any intentionally separate www or subdomain host. Read robots.txt at the origin, not beneath a content folder. Record response statuses and the final redirect destination. A browser's cached login session can conceal a failure that a public crawler encounters.

On a site you control, inspect the deployed CMS settings and confirm that proxy and CDN rules preserve the sitemap route. On a third-party site, stop after bounded public checks. A missing conventional path is not permission to brute-force endpoints or bypass access controls.