---
title: "Sitemap and robots"
url: "/help/buuk/sitemap-and-robots"
updated: "2026-10-22T11:13:50.166Z"
---
# Sitemap and robots

Crawlers need two files to index a site: robots.txt, which says what may be crawled, and sitemap.xml, which lists what should be indexed. Buuk generates both — there is nothing to configure.

## robots.txt

It is one file for the whole deployment:

- Allowed: the site root and `/help/` — every public help center.
- Disallowed: `/dashboard`, `/api`, and the sign-in and onboarding pages.
- A sitemap pointer at `/sitemap.xml`.

There is no per-workspace rule set. Crawling is permitted for the help paths, and page-level rules — such as noindex on a private center — handle everything finer than that.

## sitemap.xml

The sitemap is built from your database, not by crawling the site. For every public workspace it includes:

- The help center home
- Topic pages for each section
- Every published help article, plus guides and FAQ entries
- The 50 most recent changelog releases
- API reference operations
- Your custom tab landings when that feature is on

Article entries carry a last-modified date from update history, and a daily change frequency tells crawlers how often to return.

Only what a signed-out reader could see makes the list. Private help centers, reader-gated workspaces, and disabled platforms are skipped entirely, so gated content never appears as an indexable URL.

## Refresh

The sitemap route regenerates on each request, so publishing an article is enough for the next crawl to find it — nothing to submit or rebuild. robots.txt is stable: it changes when the app changes, not when your content does.

> [!NOTE]
> A page missing from the sitemap is usually still a draft or sits in a private section. Publish it, or check the section, and it appears on the next request.

## Related

- Showing well in chat and social: [SEO metadata and social images](./seo-metadata-and-social-images).
- The AI-facing surface: [llms.txt and markdown endpoints](./llms-txt-and-markdown).
- Naming and redirects: [Slugs and redirects](./slugs-and-redirects).
- Custom hostnames: [Custom domain](./custom-domain).