6 guides
Why these Actors do not break in three months
The decisions behind Actors that keep running: cache metadata, robots.txt compliance, concurrency limits, pay-per-event pricing and honest fill rates.
robots.txt as a spec
A site's robots.txt is the only machine-readable statement it makes about crawling. Treating it as the specification — and downloading it on every run — is why a scraper survives.
Pay-per-event pricing
Charging per delivered row changes what you are allowed to ship. If an error row costs the user money, every bug becomes a billing dispute.
Cache metadata
Shared caching makes runs fast and cheap and quietly destroys trust — unless the row itself reports where it came from and when.
Measured fill rates
A field list is a promise. A fill rate is a measurement. This is what happened when one Actor's numbers were re-measured at n=741 instead of n=30.
Naming for the event
Two Actors in this catalogue are named for what they actually measure rather than for the keyword that would sell them, and both decisions cost search volume on purpose.
Pricing a held dataset
The same answer costs $1 per thousand from a file and $20 per thousand from a WHOIS query, and the gap is a cost structure rather than a discount. What that means for how an Actor should be priced.