Design and Architecture of WOS: 58 Group's Self‑Developed Object Storage System

In recent years, the rapid growth of unstructured data such as documents, audio, and especially short videos within 58 Group’s businesses created a need for a cost‑effective, scalable storage solution, leading to the development of WOS, a self‑built object storage system offering high extensibility, low cost, strong consistency, and reliability.

The overall architecture of WOS draws inspiration from Facebook’s Haystack paper, merging many small files into large volumes while storing metadata separately, and extends support to short video files; the system is composed of several key modules.

Proxy module: Acts as the front‑end, receiving HTTP requests, parsing file metadata, and delegating storage to the Directory module. It provides two upload interfaces: for files < 4 MB, a single upload API that merges data into a large file; for files ≥ 4 MB, a multipart upload workflow (init, sliceupload, commit).

Store module: Implements the actual storage, grouping stores into groups with multiple replicated stores (typically three). Each store manages several volumes, where needles (either whole small files or fragments of large files) are packed into large volume files.

Directory module: Serves as the metadata hub, storing file information in the internal distributed KV store wtable. It performs scheduling using a weighted round‑robin algorithm based on real‑time volume metrics (capacity, latency) to assign data to appropriate stores and volumes.

Detector module: Continuously monitors the health and capacity of stores and volumes, reporting status (read/write capability, remaining space, write count, latency) to etcd, providing the data used by the Directory’s scheduler.

Fast‑upload “秒传” feature: To avoid redundant uploads, a precheck interface uploads the file’s SHA‑1 (or fragment SHA‑1s for large files). If the server finds existing identical data, it increments a reference count and returns success without transferring the data again; otherwise, the client proceeds with the normal upload flow. The server maintains three wtable tables to map IDs to SHA‑1, track reference counts, and store fragment offset information.

Erasure‑coding solution: To reduce storage cost from three‑replica to a 4+2 scheme, WOS introduces EC‑type volumes managed by groups of six stores. Encoding can be performed post‑write (after three‑replica data is written) or in‑line (during write). After testing, the in‑line encoding approach was adopted for its lower impact on write latency.

Implementation changes for EC: The Proxy now distinguishes EC volumes when reading/writing; the Directory records volume type and can schedule data to either replica or EC volumes; the Store interacts with an encodeserver to encode needles and supports data recovery; the encodeserver handles encoding of existing data and updates metadata accordingly.

In summary, WOS demonstrates 58 Group’s exploration of large‑scale unstructured data storage, continuously evolving its architecture, fast‑upload capabilities, and erasure‑coding techniques to provide a robust backend storage platform for growing business demands.