Optimizing Container Registry Storage: Introducing Blob De-Duplication for Accurate Sizing and Cost – Brandon Caton, Red Hat
Accurately calculating the true storage value of a container registry is a complex task. Enforcing quota’s correctly within a registry becomes challenging without accurate sizes, resulting in increased storage costs. In this talk, we’ll explore a new technique for determining the real storage value of a container registry – blob de-duplication. This method involves eliminating duplicate blob sizes within a namespace to provide a more accurate measure of the actual storage used by a container registry. This gives the benefits of improved cost optimization and better visibility into container registry usage. We’ll dive into the details of the implementation and our experience of deploying it at scale. By splitting the calculation in two, a backfill and running total, we are able to get accurate totals while keeping latency low. Within that running total we’re able to perform specific checks that keeps the accuracy of the de-duplicated blob total without re-calculating from scratch across millions of rows. This talk aims to inform developers, DevOp’s engineers, and operators on advancements in effective storage utilization as well as new approaches to solving complex problems.
