How to scrape coches.net car listings without breaking its robots.txt

Coches.net is the largest vehicle marketplace in Spain and, for anybody doing data work, an unusually cooperative target: it renders its listing data server-side and it states its crawl limits explicitly. Both facts shape how a scraper for it should be built, and neither is obvious until you have read the site's own files.

Why no browser is needed

Every coches.net search response already contains the full JSON payload for the listings on that page — 35 of them, with all 55 documented fields populated. Nothing is assembled by JavaScript after load.

That single property is why a run costs cents. No browser is launched, no JavaScript is executed, and one HTTP request yields 35 complete rows instead of 35 further requests to detail pages. It also means the extraction is stable in a way a browser-driven scraper is not: there is no render timing to get wrong.

What coches.net's robots.txt actually says

The file is worth reading in full before building anything against the site. The part that governs a listing scraper is the pagination block, and it is unusually explicit — rather than a wildcard, it enumerates the disallowed page parameters one by one:

Disallow: /*pg=7*
Disallow: /*pg=8*
Disallow: /*pg=9*
...
Disallow: /*pg=69*

Pages one to six are allowed; seven onward are not. That is where the Actor's maxPagesPerUrl default of 6 comes from — not from a guess about politeness, but from the site's own declaration. The same file also disallows the detail-page patterns and the internal endpoints, which is consistent with the Actor not requesting them.

Facet expansion instead of deep pagination

Six pages per URL is about 210 listings. The site holds hundreds of thousands. So how do you get coverage?

Not by paging deeper — and here is the part that makes the constraint almost irrelevant: the site's own search stops returning results past roughly page 300 anyway. Deep pagination would miss about 96% of the inventory no matter what any crawler attempted, permitted or not.

Coverage comes from breadth. Discovery expands facets — make, then province, then price band — and each combination has its own shallow, allowed page list. Thirty-four makes across fifty-two provinces is over seventeen hundred facet URLs, each yielding up to 210 listings, all within six pages.

Building the input

Every field has a default, so an empty input {} is a valid run. In practice you want to constrain it:

{
  "maxResults": 1000,
  "makes": ["bmw", "audi"],
  "provinces": ["madrid"],
  "scrapeDealers": true,
  "maxDealers": 200,
  "maxCacheAgeDays": 7
}

Make and province names come from the site's own URL scheme — bmw, mercedes-benz, madrid, a_coruna. Leave either array empty to sweep all 134 makes or all 52 provinces.

One constraint to know about before you hit it: colour filters cannot be combined with a make or badge filter. The site's URL scheme allows only one of the three, so the Actor cannot offer a combination the target does not express.

Reading the output

One row per listing, 55 fields, all from the search results. Eighteen of them are filled on 100% of a 550-listing sample — id, url, title, make, model, year, mileage, price, fuel type, province, region, seller type, photos, publication date and the boolean flags.

Enumerations are translated to English with the original preserved: fuel_type alongside fuel_type_raw, offer_type alongside offer_type_raw. Province, city and region stay in Spanish because they are proper nouns — “A Coruña” is not “The Corunna”, and translating it would break any join against another Spanish dataset. The DGT badge stays as 0, ECO, C or B for the same reason: those are official codes, not words.

The two fields worth the most are price_average_indicator_eur and price_rank_indicator: coches.net's own valuation of the model and where this listing sits against it, filled on 77.6% of listings. What they are and what they are worth.

Adding dealer profiles

Turn on scrapeDealers for a second dataset: name, contract id, phone, province, postcode, pack and active status. Measured on 60 dealers, nine of eleven fields were present on 100%, with street address and stock page URL at 96.7%.

Dealer phone numbers are returned in full because they are published business contact details. Private sellers' numbers never are, under any setting — the reasoning.

Cost and caps

$0.002 per listing, $0.004 per dealer profile, $0.00001 to start. A thousand listings is about $2 and two hundred dealers adds eighty cents. Error rows are never charged, and charges apply as each row is produced rather than in a lump at the end.

Results are capped at 50,000 per run. If you need the whole national inventory, that is several runs partitioned by facet, which is also the better shape operationally: a failure costs you one partition rather than everything.

Three gotchas

1. Detail pages are not included. Free-text descriptions, colour, transmission and equipment lists live on the individual advert pages, which sit behind stricter bot protection. They are absent from the schema rather than empty in the rows.
2. The valuation is missing on 22.4% of listings, and disproportionately on private ones — 38.2% coverage against 86.6% for dealers. A screen built on it is implicitly a dealer screen unless you handle the null.
3. Cached rows are cheap and old. Fine for depreciation analysis, not fine for “is this car still available”. Set maxCacheAgeDays: 0 when freshness is the point.