How we scaled metrics aggregation by moving durable state into storage
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Datadog engineer Khayyam Guliyev describes three generations of its metrics aggregation service, each built for roughly 10× more volume. Static partitions with local RocksDB state doubled compute and needed manual splits. Dynamic Kafka assignment with blob-backed state made recovery the bottleneck. The third generation makes streaming best-effort and has storage reconcile partial aggregates, duplicates and late data, so workers resume at the live edge. The figures are Datadog's own, unsourced, and the evidence for the recovery benefit is one production incident.