Our S3 cache is full of holes – We put them there
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
The post explains how Berserk, a log, metrics, trace and eval data store, uses sparse files on Linux as an S3 cache. Query nodes store only needed 4 MiB blocks, punch holes with fallocate to reclaim disk during eviction, and coordinate with query workers through a shared presence bitmap and…