How to Extract Sitemap URLs to CSV or JSON

For a urlset, extract loc directly beneath each url. An index's loc values point to files, not pages; follow them with a visited set and safety limits. Do not collect every loc indiscriminately: extension elements may contain image or video URLs that are not page entries. Keep the traversal manifest if you need to account for every child.

Protocol-compliant loc values must be absolute HTTP(S) URLs. SitemapKit extraction tolerates relative values by resolving them against the fetched response URL and removes fragments; Checker can still flag the source loc as invalid. Successful extraction is not a protocol certificate. Retain raw values separately when diagnosing source data.

Page URL deduplication uses URL parsing with fragments removed. It does not equate HTTP with HTTPS, www with apex page URLs, trailing-slash variants, or reordered query strings. Do not strip parameters or lowercase paths blindly: those changes can alter the resource. Let canonical checks decide which variants belong in the audit set.

The extract API returns loc and optional lastmod, changefreq, and priority for each entry. It does not return per-entry source_sitemap, HTTP status, canonical URL, or indexability. The columns below are useful for an enriched audit, not a claim about the API response. Extract children individually and attach their requested locations when provenance matters; collect page diagnostics separately.

Read SitemapKit entries from data.urls, with loc as the URL field. Preserve meta and data-level completion flags alongside the export so downstream jobs can identify a bounded or interrupted inventory. The following is an illustrative entry, not a captured response; optional metadata is omitted when absent.

POST to /api/v1/sitemap/extract with application/json and a valid x-api-key header. A valid first-party session is also accepted when no key is supplied; scripts should use a key. An invalid supplied key does not fall back to a session. Keep keys out of exports and client-side code.

maxUrls must be an integer from 1 to 50,000 and defaults to 50,000. The effective allowance is also capped by the plan and admitted quota. Begin with a modest explicit budget, then raise it when you know the inventory size and available allowance. Replace the example target below with a verified public sitemap; example.com is not promised to publish one.

HTTP 200 successes have success, data, and meta. data includes sitemapUrl, urls, totalUrls, sitemapsProcessed, and truncated; partial and stopReason may be added. Check meta.partial too. Cached successes have meta.cached true and creditsUsed 0, but omit the live response's rateLimit and quota objects. Do not require those objects in every successful response.

Direct extraction visits at most 50 sitemap locations including the starting index, with traversal capped at depth 5 and a foreground deadline. stopReason can identify max_urls, sitemap_limit, deadline, or aborted. There is no pagination cursor: repeating the same oversized index does not retrieve a next batch. Extract known children separately instead.

totalUrls counts unique entries returned, not the site's total eligible pages. Individual failed child fetches can be skipped without setting partial. Thus partial false is not proof of complete traversal. Compare expected children and their counts, test missing files directly, and reconcile against the CMS inventory before using the result for a migration or deletion job.

Inspect the status and JSON code before reading data. UNAUTHORIZED is 401; malformed JSON or invalid input is 400. RATE_LIMIT_EXCEEDED, CONCURRENCY_LIMIT_EXCEEDED, and MONTHLY_QUOTA_EXCEEDED are distinct 429 outcomes. On a rate-limit response, Retry-After is a delay in seconds, while JSON retryAfter is the reset timestamp. A monthly quota failure should not trigger a tight retry loop.

Uncached successes charge at least one credit, then one per started 1,000 returned URLs. Cached successes use zero credits but still pass authentication and request-rate checks. Quota is reserved before work and settled against returned entries; available quota can lower the admitted URL budget. Read meta.creditsUsed and, when present, meta.quota rather than assuming one request always costs one credit.

Retain X-Request-Id for failed calls. Capacity or tracking failures may return 503, upstream failures 502, and timeouts 504. An error body is not an export. A cached result can also predate the site's newest publication, so record the time and cached flag with the inventory.

Write UTF-8, quote fields that contain commas or line breaks, and neutralize cells that begin with spreadsheet formula characters when exporting untrusted values. Stream large exports rather than building the whole file in memory.

Once exported, compare the sitemap set with analytics landing pages, internal crawl URLs, and the CMS inventory. The differences reveal orphan pages, stale URLs, and pages that are internally linked but missing from the sitemap.