SitemapKit + LlamaIndex

Prerequisites

• A SitemapKit API key kept outside browser and mobile client bundles.
• A server-side LlamaIndex environment that can make HTTPS requests and parse JSON responses.

How to connect SitemapKit and LlamaIndex

1. 1

   Store the API key for LlamaIndex

   Use the secret or environment-variable mechanism provided by your ai frameworks setup. Never expose the key in client-side code.

2. 2

   Call the SitemapKit full endpoint

   Adapt the request shown in the LlamaIndex example below and send the target domain from a trusted server process.

3. 3

   Map the response in LlamaIndex

   Handle discovered sitemap files, extracted URLs, lastmod values, truncation flags, and API errors before passing the result downstream.

Quick start

import requests
from llama_index.core import VectorStoreIndex
from llama_index.readers.web import SimpleWebPageReader

# Get all URLs from sitemap
resp = requests.post(
    "https://sitemapkit.com/api/v1/sitemap/full",
    headers={"x-api-key": "YOUR_API_KEY", "Content-Type": "application/json"},
    json={"url": "docs.example.com"}
)
urls = [u["loc"] for u in resp.json()["urls"]]

# Load and index pages
documents = SimpleWebPageReader(html_to_text=True).load_data(urls[:100])
index = VectorStoreIndex.from_documents(documents)

# Query the index
query_engine = index.as_query_engine()
response = query_engine.query("How do I configure authentication?")
print(response)

Continue with SitemapKit