Most scraping guides mention robots.txt once, in a paragraph about ethics, and then never again. That framing is why it gets ignored: presented as a moral obligation it competes with getting the job done. Presented as what it actually is — the only machine-readable specification the site publishes about automated access — it stops being a competing concern and becomes the document you design from.
What robots.txt is and is not
It is a plain-text file at a site's root in which the site states which URL patterns automated clients may and may not request. It is a declaration, not a technical control: nothing enforces it, which is exactly why respecting it is a decision rather than a constraint.
It is not a statute, and its legal weight varies by jurisdiction and context. What it does unambiguously establish is intent — which converts ignoring it from an oversight into a deliberate act, and that distinction tends to matter more than people expect when anything goes wrong.
Why the file gets downloaded every run
The tempting shortcut is to read it once during development, encode the rules as constants, and ship. It works until the site changes its mind — and then your scraper is confidently violating rules it has never seen, using a copy that was accurate in August.
Downloading it live at the start of every run costs one HTTP request and makes compliance a property of the code rather than a claim in a README. If coches.net tightens its pagination rules tomorrow, the next run adapts; nobody has to notice, remember, and deploy.
Reading a real one
Here is the part of coches.net's file that governs a listing scraper. It is not a wildcard — it enumerates the disallowed pagination values one at a time:
User-agent: *
...
Allow: /concesionarios/
Disallow: /*/Detail/
Disallow: /ws/
Disallow: /*pg=7*
Disallow: /*pg=8*
...
Disallow: /*pg=69*
Disallow: /*?q=*Four design decisions fall straight out of that:
- Dealer pages are explicitly allowed, so a dealer dataset is on the intended surface.
- Detail-page patterns are disallowed, which is one of two reasons the Actor does not request advert pages — the other being that they sit behind stricter bot protection.
- Internal endpoints under
/ws/are off limits, so no “undocumented API” shortcut. - Pagination stops at six. Which determines the entire discovery strategy.
Pagination limits hide in plain sight
A pagination disallow is the most consequential rule a listing site can publish and the easiest to miss, because it looks like clutter. Six allowed pages is roughly 210 rows per URL — nowhere near an inventory of hundreds of thousands.
The answer is breadth rather than depth: expand facets — make, then province, then price band — so each combination has its own shallow, permitted page list. How that works in practice.
When the rules and your interests agree
Far more often than the ethics framing suggests. On coches.net, deep pagination is disallowed and ineffective: the site's own search stops returning results past roughly page 300, so a crawler that ignored the rule would still miss about 96% of the inventory. The compliant strategy is also the only one that works.
The same pattern shows up in rate limiting. Measured against Nextdoor, concurrency 5 ran clean while 10 triggered a roughly four-minute cooldown — so the polite configuration is the fast one, and the aggressive one is slower in wall-clock terms. Sites are not usually punishing you arbitrarily; they are protecting infrastructure whose failure modes also degrade your run.
When they do not
Sometimes the rules simply put the data you want out of reach. Then the honest outcome is that the data is unavailable to you by that route, and the correct behaviour for the scraper is to report the failure rather than route around it.
Concretely, that means no CAPTCHA solving, no TLS or browser fingerprint spoofing, and no requesting a path the file disallows. It is a real product limitation and it is stated on each Actor's page as one — because a tool whose reliability comes from evading detection has a reliability that lasts exactly until the detection improves.
