ActorStack.dev
Scraper engineering

llms.txt

A plain-text file at a site's root that describes the site, its scope and its canonical URLs for the benefit of language models rather than search crawlers.

It is a convention rather than a standard, and its value is in stating the things a model would otherwise guess: what the source is, how it should be cited, and what it does not cover.

Where this comes up