Birta
← Back to Birta

Glossary

What is robots.txt?

robots.txt is a file at a site's root that tells crawlers what they may fetch, following a real internet standard (RFC 9309) well-behaved bots are expected to obey.

robots.txt is a plain-text file at a site's root that tells a crawler what it may fetch. It follows RFC 9309, a real internet standard, and well-behaved crawlers are expected to obey it, though nothing forces a bot to.

It answers a different question than llms.txt: robots.txt is a rule about access, not a curated summary of what a site is. Birta's own robots.txt points crawlers at both /llms.txt and /llms-full.txt.

For how the two files differ and what each is actually for, see what is an llms.txt file, a real one explained line by line.