robots.txt
A plain-text file at a site's root in which the site declares, in machine-readable form, which URL patterns automated clients may and may not request.
It is not a statute, but it is an explicit statement of the site's wishes, which makes ignoring it deliberate rather than accidental. Downloading it live on every run means a rule change takes effect immediately instead of at the next deploy.