The story’s score and claims

Comments on No. 19 of 142 · Datadog Engineering · edition of Wednesday 18 November 2026

How we scaled metrics aggregation by moving durable state into storage

Summary

Datadog engineer Khayyam Guliyev describes three generations of its metrics aggregation service, each built for roughly 10× more volume. Static partitions with local RocksDB state doubled compute and needed manual splits. Dynamic Kafka assignment with blob-backed state made recovery the bottleneck. The third generation makes streaming best-effort and has storage reconcile partial aggregates, duplicates and late data, so workers resume at the live edge. The figures are Datadog's own, unsourced, and the evidence for the recovery benefit is one production incident.

Comments

Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules