Sitemap API for Web Scraping

Before you start

• A SitemapKit API key stored in a server-side environment variable.
• A target domain or sitemap URL and a clear success criterion for the web scraping workflow.

Implementation workflow

1. 1

   Define the workflow objective

   Start from this concrete goal: Replace full-site crawls with sitemap extraction

2. 2

   Request the sitemap data

   Use the implementation example below from server-side code and pass the target domain or sitemap URL required by the workflow.

3. 3

   Validate and reuse the result

   Check the returned sitemap URLs and page records against the expected outcome for web scraping before sending the data to the next system.

Why use SitemapKit?

• Get all URLs in one API call instead of crawling page by page
• 10-100x faster than a full site crawl
• Respectful to target servers — no crawl pressure
• Includes lastmod to skip unchanged pages

Implementation example

import requests

# Get all product URLs from an e-commerce site
resp = requests.post(
    "https://sitemapkit.com/api/v1/sitemap/full",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"url": "shop.example.com"}
)

urls = resp.json()["urls"]
product_urls = [u["loc"] for u in urls if "/products/" in u["loc"]]
print(f"Found {len(product_urls)} product pages to scrape")

Continue with SitemapKit