A web crawler is a bot that fetches pages automatically, usually to build a
search index or gather data for some other system. It follows links from
page to page on its own, without a person directing each request, and reads
robots.txt to learn which paths it is invited to fetch.
Most requests for a file like llms.txt turn out to come from crawlers of
this kind, not from AI assistants answering a specific question.
Ahrefs' own analysis found that
SEO audit tools and general crawlers made up most of the traffic to the
llms.txt files it studied.
For the full breakdown of who actually requests these files, see what is an llms.txt file, a real one explained line by line.