Optimizing Kuaishou's Photo Object Storage: Reducing Size and Boosting Cache Hit Rate

Kuaishou, a national short‑video platform, stores billions of videos whose core metadata is encapsulated in the "Photo" object, a relational‑database entity containing over 200 attributes such as title, bitrate, cover, timestamps, location, and commercial information.

The name "Photo" originates from the platform’s early "Gif Kuaishou" era, where short animated GIFs were treated as photos; the name persisted after the product shifted to short videos and live streaming.

Because Photo objects are accessed by thousands of upstream services and cached at massive scale (hundreds of GB to TB of cache, billions of QPS), Kuaishou employed database sharding with read‑write separation, but this high availability came at the cost of massive storage redundancy.

Key optimization results include:

Storage space reduced by ~25%, saving several hundred terabytes of disk.

Average size of newly created Photo objects decreased by ~1 KB, cutting daily DB growth by hundreds of gigabytes.

Cache hit rate rose from 95 % to 97 %, translating to millions of dollars in annual cost savings.

The Photo table’s JSON‑encoded extension field grew from ~60 attributes in 2019 to over 230 in 2023, with the top 30 attributes consuming 93 % of space and many holding default values (0, false, null). By adding the Jackson annotation @JsonInclude(Include.NON_DEFAULT) to these fields, default values are omitted during serialization, shrinking each object by roughly 1 KB.

For existing data, a two‑month cleaning campaign used a 16‑core container with hundreds of threads to iteratively read, trim, and rewrite records across 100 database shards, carefully throttling the operation to avoid binlog lag and downstream overload. This process reclaimed ~400 TB of space and created “holes” that future inserts could fill.

Attempts to further compress tables via ALTER ENGINE=InnoDB yielded mixed results; some shards saw space reductions while others experienced growth due to page fragmentation, a bug later confirmed by MySQL developers.

Beyond the technical fixes, the team reflected on broader governance topics: aligning data‑storage costs with business models, timing architectural upgrades, and choosing appropriate data encodings (JSON vs. Protobuf) based on edit frequency and concurrency.

In summary, a modest engineering effort—targeted field pruning, selective serialization, and disciplined data cleaning—delivered outsized benefits across storage, network bandwidth, cache efficiency, and overall system stability, illustrating the value of continual low‑level optimization in large‑scale UGC platforms.