ActorStack.dev
Scraper engineering

Pagination depth

How many pages deep into a result list a crawler goes before it stops, often capped both by the site's robots.txt and by the point at which the site stops returning results.

coches.net disallows pagination parameters from `pg=7` upward, and its search stops producing results past roughly page 300 in any case. When the allowed depth is shallow, coverage has to come from breadth.

Where this comes up