Sitemap Crawler

Crawl any domain to map its complete sitemap structure. See every sitemap with per-sitemap URL counts, aggregate stats, and download options.

What is a Sitemap Crawler?

A sitemap crawler automatically discovers and processes all XML sitemaps on a domain. It checks the robots.txt file for sitemap declarations and probes common sitemap paths like /sitemap.xml and /sitemap_index.xml.

For each discovered sitemap, the crawler counts the number of URLs it contains. This tree view helps you understand how a website organizes its content across multiple sitemaps — useful for SEO audits and competitive analysis.

How the Sitemap Crawler Works

1

Discover sitemaps

We check robots.txt and probe common paths to find every sitemap on the domain.

2

Crawl each sitemap

We fetch and parse each discovered sitemap, counting the URLs it contains and following any sitemap index references.

3

Build the tree

We present a tree view showing each sitemap with its URL count, plus aggregate totals and download options.

Frequently Asked Questions

What does a sitemap crawler do?

A sitemap crawler discovers all sitemaps on a domain by checking robots.txt and common sitemap paths, then crawls each sitemap to count the URLs it contains. This gives you a complete overview of a site's sitemap structure.

How is this different from a sitemap finder?

A sitemap finder discovers which sitemaps exist on a domain. The sitemap crawler goes further — it processes each discovered sitemap and reports per-sitemap URL counts, giving you a tree view of the entire sitemap structure.

Can I download the results?

Yes. You can download the sitemap tree as a CSV file (with URL counts per sitemap), a plain text file (sitemap URLs only), or a JSON file with the complete data.

How many sitemaps can it crawl?

The tool processes up to 50 sitemaps per domain and extracts up to 5,000 URLs per sitemap for counting purposes. For larger sites, sign up for a free API key.

Need full sitemap data at scale?

Get a free API key to discover and extract all URLs from any domain. 100 requests/month included.