Sitemap API for LLM Agents & RAG Pipelines

Before you start

• A SitemapKit API key stored in a server-side environment variable.
• A target domain or sitemap URL and a clear success criterion for the llm agents & rag pipelines workflow.

Implementation workflow

1. 1

   Define the workflow objective

   Start from this concrete goal: Feed URL lists to AI agents and RAG pipelines

2. 2

   Request the sitemap data

   Use the implementation example below from server-side code and pass the target domain or sitemap URL required by the workflow.

3. 3

   Validate and reuse the result

   Check the returned sitemap URLs and page records against the expected outcome for llm agents & rag pipelines before sending the data to the next system.

Why use SitemapKit?

• Get all URLs on a domain without expensive full-site crawling
• Perfect seed URLs for RAG indexing pipelines
• Works with LangChain, LlamaIndex, CrewAI, and custom agents
• llms.txt compatible — agents can discover the API automatically

Implementation example

import requests

# Step 1: Get all URLs from sitemap
response = requests.post(
    "https://sitemapkit.com/api/v1/sitemap/full",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"url": "docs.example.com"}
)
urls = [u["loc"] for u in response.json()["urls"]]

# Step 2: Feed to your RAG pipeline
for url in urls:
    content = crawl_page(url)  # Crawl4AI, Firecrawl, etc.
    index.add(content)         # Add to vector store

Continue with SitemapKit