Site files from one config
robots.txt, llms.txt, sitemap.xml, rss.xml, security.txt and humans.txt – configured with types and rewritten on every astro build. No hand-editing in public/.
One integration instead of three plugins and manual edits.
astro-site-files writes all of a site’s meta files to the build output in astro:build:done. robots.txt and sitemap.xml are on by default; the other four files are generated once their option is set. The package replaces @casoon/astro-crawler-policy and @casoon/astro-sitemap, neither of which is actively maintained any more.
- 01Typed — TypeScript interfaces for every option, mistakes show up in the editor
- 02Build time — no runtime code, no external service
- 03Presets — pick a crawler strategy by name instead of listing bots by hand
- 04Audit hints — warnings about misconfiguration before the files go live
Six files, each one optional.
Crawler rules
Presets, group and bot overrides, custom agent blocks and a Sitemap: line that points at the file actually written.
Sitemap
Built from the build output, with dynamic sources, priority and changefreq rules, hreflang and index mode from 50,000 URLs.
Overview for AI systems
Title, description and link sections following llmstxt.org. Sections can be generated from content collections via sources.
RSS 2.0 feed
Top-level rss option, independent of the sitemap. For SSR and development there is also createRssRoute from the /rss sub-path.
Security contact
Contact, Expires, Policy, Encryption and more. Expired or invalid dates are reported as errors by the audit hints.
Team and technology
Team, thanks, technologies and a note following humanstxt.org. The last-update date defaults to the build date.
One config, three results.
// astro.config.ts
import { defineConfig } from 'astro/config'
import siteFiles from '@casoon/astro-site-files'
export default defineConfig({
site: 'https://example.com',
integrations: [
siteFiles({
robots: { preset: 'seoOnly', disallow: ['/admin/'] },
llms: {
title: 'Example',
description: 'An example website focused on TypeScript tooling.',
details: 'This site documents internal tools and workflows.',
sections: [{
title: 'Documentation',
links: [
{ title: 'Getting started', url: '/docs/start', description: 'Setup guide' },
{ title: 'API reference', url: '/docs/api' },
],
}],
},
security: {
contact: 'mailto:security@example.com',
expires: '2027-01-01T00:00:00.000Z',
},
}),
],
})User-agent: *
Disallow: /admin/
User-agent: Googlebot
Allow: /
Disallow: /admin/# Example
> An example website focused on TypeScript tooling.
This site documents internal tools and workflows.
## Documentation
- [Getting started](/docs/start): Setup guide
- [API reference](/docs/api)Five presets, five bot groups.
| Bot group | seoOnly | citationFriendly | openToAi | blockTraining | lockdown |
|---|---|---|---|---|---|
| Search engines | allow | allow | allow | allow | disallow |
| Verified AI bots | disallow | allow | allow | allow | disallow |
| Unknown AI scrapers | disallow | disallow | allow | disallow | disallow |
| SEO scanners | inherit | inherit | inherit | inherit | disallow |
| Web archives | disallow | inherit | allow | disallow | disallow |
| Training bots also blocked | – | yes | – | yes | – |
As of v0.5.0, September 2026, per the README. inherit means no group of its own, the wildcard group applies. Highlighted is seoOnly, the preset used in the quick start. groups and bots override any preset.
What exactly happens in the build output.
Version 0.5.0 changed several rules that apply without any new configuration.
- 01
Global rules in every bot group
Under RFC 9309 a crawler obeys only its most specific matching group. So global
disallowpaths are repeated in every allowed bot block. Blocked bots keep a bareDisallow: /. - 02
noindex pages leave the sitemap
Pages with
<meta name="robots" content="noindex">are no longer listed in sitemap.xml; Search Console would report them as errors. Opt out withexcludeNoindex: false. - 03
hreflang from Astro’s i18n
sitemap.i18ndefaults to Astro’s i18n config and writes alternates includingx-default. Locale prefixes no longer lower the priority. - 04
Values from JSON-LD
data-sitemap-priorityanddata-sitemap-changefreqon JSON-LD script tags, for example from@casoon/astro-structured-data, are applied per page.
Release history
Shared audits
Unified audit types, llms.sources, sitemap deduplication.
RSS
RSS 2.0 feed and createRssRoute taken over from @casoon/astro-sitemap.
Presets and registry
Bot registry and the five presets ported from @casoon/astro-crawler-policy.
Behaviour changes
noindex filter, hreflang from Astro i18n, rss as its own option, new security.txt checks.
What these files cannot do.
robots.txt is voluntary
Well-behaved crawlers follow it. Scrapers with regular browser headers and rotating IPs ignore it. A block can only be enforced at server or CDN level.
Read before upgrading
0.5.0 changes sitemap and robots.txt output without a config change. Check the changelog and diff the generated files before you update.
RSS without getCollection
getItems runs in astro:build:done, where getCollection() is not available; entries are read from the file system.
sources are not filtered
exclude and filter apply only to pages discovered automatically. Entries from sources have to be filtered in the function itself or via serialize.
Frequently asked questions.
Which files does astro-site-files generate?
robots.txt, llms.txt, sitemap.xml (or sitemap-index.xml with numbered chunks), rss.xml, .well-known/security.txt and humans.txt. robots.txt and sitemap.xml are on by default; the others are generated once their option is set.
Does astro-site-files replace the older CASOON packages?
Yes. It is the successor to @casoon/astro-crawler-policy (robots.txt and llms.txt) and @casoon/astro-sitemap (sitemap.xml and rss.xml). Neither is actively maintained any more; new features and fixes only land in astro-site-files.
How do I choose a crawler strategy?
With a preset name, for example robots: { preset: "seoOnly" }. The options are seoOnly, citationFriendly, openToAi, blockTraining and lockdown. groups adjusts a whole bot group, bots a single bot; extraBots adds your own.
Does robots.txt reliably keep AI crawlers out?
No. robots.txt is a statement of intent. Crawlers that honour it will respect it; many scrapers do not identify themselves as bots. Effective blocking needs rate limiting, bot detection or WAF rules at server or CDN level.
What are the requirements?
Node.js 22.12.0 or later and Astro 6.0.0 or later, including Astro 7. The renderer functions such as renderRobotsTxt and renderSitemapXml are exported as well and can be used without the Astro integration.
When are the files written?
During the build, in the astro:build:done hook. All files go straight into the build output. No separate step and no running server are needed.
One package for all site files.
Configure once, regenerate on every astro build.