From 6 Hours to 1 Minute: Checkpointing Kubernetes-Style Controllers

Confluent describes a checkpointing framework for its Kubernetes-style controllers, which run on an in-house Resource Platform rather than etcd. Controllers periodically save long-lived watch continue tokens (about a week), unfinished queue requests and optional extra state, then resume from them instead of re-listing everything. Confluent reports its largest controller restarting in 1 minute instead of 6 hours, against 25 million objects and 115 GB in its biggest region. These figures are self-reported, and the checkpoint is skipped when the queue is too large.