4 guides
Why these Actors do not break in three months
The decisions behind Actors that keep running: cache metadata, robots.txt compliance, concurrency limits, pay-per-event pricing and honest fill rates.
robots.txt as a spec
A site's robots.txt is the only machine-readable statement it makes about crawling. Treating it as the specification — and downloading it on every run — is why a scraper survives.
Pay-per-event pricing
Charging per delivered row changes what you are allowed to ship. If an error row costs the user money, every bug becomes a billing dispute.
Cache metadata
Shared caching makes runs fast and cheap and quietly destroys trust — unless the row itself reports where it came from and when.
Measured fill rates
A field list is a promise. A fill rate is a measurement. This is what happened when one Actor's numbers were re-measured at n=741 instead of n=30.