NebulasFs: A Distributed High‑Availability Small‑File Storage System
NebulasFs is a self‑developed distributed, highly available and durable storage system designed to efficiently store billions of small files by using a master‑datanode architecture, multi‑tenant isolation, customizable replication, automatic scaling, and automated replica repair, addressing the challenges of massive unstructured data generated by modern applications.
In response to rapid business growth and the explosion of unstructured media data, 360's infrastructure team built NebulasFs, a distributed small‑file storage system capable of handling hundreds of billions of files.
Inspired by Facebook's Haystack, NebulasFs provides a high‑availability, high‑reliability, persistent storage solution with a master‑datanode architecture; the master manages metadata, cluster management, and scheduling, while datanodes handle storage, routing, and proxy functions.
The system offers a simple HTTP RESTful API for PUT/GET operations, full request proxying with automatic routing, and multi‑tenant resource pools that isolate workloads across physical hardware.
It supports customizable multi‑replica strategies across five defined fault domains (data center, rack column, rack, machine, disk) to ensure data durability and availability under various failure scenarios.
NebulasFs includes automated capacity expansion and rebalancing, as well as automatic replica repair coordinated by the master, minimizing manual intervention during node failures.
After nearly a year of internal use, NebulasFs also serves as backend storage for an object storage service compatible with the AWS S3 protocol, and will continue to evolve alongside growing business demands.
360 Tech Engineering
Official tech channel of 360, building the most professional technology aggregation platform for the brand.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.