Sitemap Glossary

What is robots.txt?

A text file at the root of a website that tells search engine crawlers which pages to crawl or avoid.

Written by SitemapKit Editorial TeamUpdated: Jul 21, 2026

Key takeaways

Interpret robots.txt in its sitemap XML context; the detailed definition below explains its role and constraints.
Validate robots.txt against a real XML example instead of relying on the field or tag name alone.

The robots.txt file is a plain text file located at the root of a website (e.g., `https://example.com/robots.txt`) that provides instructions to web crawlers about which parts of the site they should or shouldn't access.

It uses the Robots Exclusion Protocol and can include directives like `User-agent`, `Disallow`, `Allow`, `Crawl-delay`, and importantly, `Sitemap`. The `Sitemap:` directive tells crawlers where to find the site's XML sitemap.

This is one of the primary ways search engines discover sitemaps. SitemapKit's discover endpoint checks robots.txt as its first step when finding sitemaps for a domain.

XML example

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-news.xml

Continue with SitemapKit

Check an XML sitemap

Inspect robots.txt in a real sitemap.

Extract sitemap URLs

Turn XML sitemap fields into structured, exportable URL data.

API documentation

See how this concept appears in SitemapKit API responses.

Practical sitemap workflows

Apply sitemap concepts to SEO, monitoring, and data pipelines.

Sources and further reading

Work with sitemaps programmatically

SitemapKit's API discovers and parses XML sitemaps into structured JSON, including fields such as robots.txt.

Get Free API Key API Docs