Sitemap API for AI Training Data Collection

Before you start

• A SitemapKit API key stored in a server-side environment variable.
• A target domain or sitemap URL and a clear success criterion for the ai training data collection workflow.

Implementation workflow

1. 1

   Define the workflow objective

   Start from this concrete goal: Build URL inventories for AI training pipelines

2. 2

   Request the sitemap data

   Use the implementation example below from server-side code and pass the target domain or sitemap URL required by the workflow.

3. 3

   Validate and reuse the result

   Check the returned sitemap URLs and page records against the expected outcome for ai training data collection before sending the data to the next system.

Why use SitemapKit?

• Discover all content URLs on any domain instantly
• Filter by content type using URL patterns
• Use lastmod to prioritize fresh content
• Scale across thousands of domains

Implementation example

import requests
import json

domains = ["docs.python.org", "react.dev", "developer.mozilla.org"]
all_urls = []

for domain in domains:
    resp = requests.post(
        "https://sitemapkit.com/api/v1/sitemap/full",
        headers={"x-api-key": "YOUR_API_KEY"},
        json={"url": domain}
    )
    urls = resp.json().get("urls", [])
    all_urls.extend([u["loc"] for u in urls])

print(f"Collected {len(all_urls)} URLs for training data")
with open("training_urls.json", "w") as f:
    json.dump(all_urls, f)

Continue with SitemapKit