Shared caching is how a scraper becomes cheap and fast. It is also how a dataset quietly becomes undatable: a row that came from a cache looks exactly like a row that came from the live site, unless the row says which it was.
The problem with an invisible cache
Caching itself is entirely legitimate. If someone else asked for the same business an hour ago, fetching it again wastes a request and produces the same answer. Serving the cached copy is the correct engineering decision.
What is not correct is doing it silently. A user screening for cars priced below market is making a decision that depends on the price being current, and a six-day-old row is a car that may have sold five days ago. Without a marker on the row there is no way to tell — and, worse, no way to know that there is something to check.
The three fields
| Field | What it tells you |
|---|---|
from_cache | Whether this row was fetched during your run or served from the shared cache. |
fetched_at | When the data actually came off the target site. |
data_age_hours | How old the data is, so you do not have to do the subtraction in every query. |
The third is redundant on purpose. It is derivable from the second, and putting it in the row means a freshness condition is a comparison rather than a date calculation — which is the difference between people filtering on age and people meaning to.
The coches.net Actor adds scraped_at alongside fetched_at: when your run produced the row, versus when the data came off the site. Two timestamps because they answer two questions, and only one of them is about whether the price is current.
Why it belongs on the row, not the run
A single run routinely mixes fresh and cached rows — some businesses were in the cache, others were not. A run-level note (“this run used caching”) is therefore true and useless: it cannot tell you which rows it applies to.
Row-level metadata also survives the journey. Rows get exported, joined, appended to a warehouse table and read six months later by somebody who has never seen the run that produced them. A field on the row is still there; a note in a run log is not.
How much age is acceptable
Entirely dependent on the field, which is why the tolerance is the user's to set:
- A business phone number changes rarely. A week-old row is fine; a month-old row is probably fine.
- A car's asking price changes constantly, and the car itself disappears. Hours matter.
- City demographics barely move year to year. Cache them aggressively.
- Anything you are about to act on individually — call, email, buy — should be fresh, regardless of the field.
maxCacheAgeDays exists so this is a decision you make rather than one made for you. Default 7, which suits the majority of analytical use.
Forcing a fresh read
maxCacheAgeDays: 0 and every row is read from the target during your run. It costs more wall-clock time and puts more load on the site, which is precisely why it is not the default — but it is the right setting for a live bargain hunt, a contact list you are about to work, or any snapshot that will become a point in a time series.
-- Only act on rows read in the last two days
SELECT ad_id, url, price_eur, data_age_hours
FROM listings
WHERE data_age_hours < 48
ORDER BY price_eur ASC;That condition is only possible because the field is there. The general principle is worth more than the three fields: whatever your pipeline is going to have to reason about later, put it on the row while you still know it.
