SitemapKit + Crawl4AI

Prerequisites

• A SitemapKit API key kept outside browser and mobile client bundles.
• A server-side Crawl4AI environment that can make HTTPS requests and parse JSON responses.

How to connect SitemapKit and Crawl4AI

1. 1

   Store the API key for Crawl4AI

   Use the secret or environment-variable mechanism provided by your ai frameworks setup. Never expose the key in client-side code.

2. 2

   Call the SitemapKit full endpoint

   Adapt the request shown in the Crawl4AI example below and send the target domain from a trusted server process.

3. 3

   Map the response in Crawl4AI

   Handle discovered sitemap files, extracted URLs, lastmod values, truncation flags, and API errors before passing the result downstream.

Quick start

import requests
from crawl4ai import AsyncWebCrawler

# Step 1: Get all URLs via SitemapKit
resp = requests.post(
    "https://sitemapkit.com/api/v1/sitemap/full",
    headers={"x-api-key": "YOUR_API_KEY", "Content-Type": "application/json"},
    json={"url": "docs.example.com"}
)
urls = [u["loc"] for u in resp.json()["urls"]]
print(f"Found {len(urls)} URLs to crawl")

# Step 2: Crawl with Crawl4AI for LLM-ready content
async with AsyncWebCrawler() as crawler:
    for url in urls[:50]:
        result = await crawler.arun(url=url)
        if result.success:
            # result.markdown contains clean, LLM-ready text
            save_to_dataset(url, result.markdown)

Continue with SitemapKit