Technical methodology

How SitemapKit collects and presents sitemap data

This page documents the collection boundaries behind the public tools, API, and domain directory so results can be interpreted and reproduced accurately.

1. Normalize the submitted target

The service accepts an HTTP or HTTPS website target, normalizes its origin, rejects unsafe destinations, and applies request budgets before any discovery work begins.

2. Discover public sitemap candidates

Discovery checks sitemap directives in robots.txt and a bounded list of common public paths. Responses are validated as sitemap XML rather than accepted only because a filename ends in .xml.

3. Traverse indexes with limits

Sitemap indexes are followed recursively with depth, concurrency, response-size, URL-count, and time limits. Entries are normalized and deduplicated while preserving available metadata such as lastmod.

4. Store successful directory snapshots

Public directory pages render the latest successful stored snapshot. A page request never waits for a new external crawl; scheduled background work refreshes data separately.

5. Publish transparent timestamps

Directory pages display the absolute crawl date. Sitemap lastmod values come from content or snapshot update dates rather than being reset across every URL during a deployment.

Important interpretation limits

— Results describe publicly accessible sitemap infrastructure at the recorded time.
— A missing sitemap does not prove that a site is unindexable.
— URL counts can be truncated when a documented safety limit is reached.
— Sitemap presence and lastmod values do not guarantee search engine indexation.
— Rankings and traffic are never inferred from sitemap data alone.

Corrections and updates

Stored snapshots are refreshed by background jobs. When the public source changes, the next successful collection replaces the data shown on the directory page. Editorial corrections update the visible review date and the corresponding sitemap lastmod value.