Seven Key Aspects of Distributed Storage Systems

Motivation – The storage domain can be organized into seven aspects: replication, storage engine, transactions, analytics, multi‑core, computation, and compilation.

Distributed Storage – Any storage system (object, block, file, KV, log, OLAP, OLTP) that performs partitioning and replication across multiple machines qualifies as distributed storage.

1. Replication

Failure detection and lease protocols.

Leader election, primary‑uniqueness invariants, network partitions, split‑brain, Byzantine faults, fail‑fast/stop, failover.

Log replication, replicated state machines.

Membership or configuration changes, host up/down management, scaling.

Data replication, rebalancing, recovery.

Replica placement and routing logic.

Fencing between primary and backup.

External consistency, linearizability.

Protocols such as pipeline, fan‑out, quorum‑based, gossip.

Distributed logging, active‑standby and active‑active architectures.

2. Storage Engine

The engine refers to the local persistent storage engine, which must balance CPU, memory, and device bandwidth/latency. Its design can be summarized as “1‑3‑5”.

1 – Calls to fsync (or equivalent), their frequency and data volume, and how to amortize them across I/O.

3 – Trade‑offs among read, write, and space amplification.

5 – The five LSNs of a write‑ahead log: prepare point, commit point, apply point, checkpoint, prune point.

Key points include group commit between prepare and commit, visibility of committed data after apply, checkpointing for durable storage, crash recovery from the latest checkpoint to the commit point, and log truncation at the prune point.

Data Structures and Algorithms – Efficient in‑memory and on‑disk structures, compression, and encoding are essential for managing storage.

3. Transactions

Transactions provide ACID guarantees; they serve as a baseline for evaluating a storage system’s correctness and concurrency handling.

How to resolve conflicts between concurrent transactions (read‑write, write‑read, write‑write): lock‑wait vs. abort‑and‑retry.

Visibility of committed transaction effects to outstanding reads.

Ideal correctness assumes serial execution; ideal concurrency assumes no conflicts. Real systems must insert synchronization points to resolve conflicts, using lock‑based (pessimistic) or timestamp‑ordering (optimistic) concurrency control.

Distributed transactions involve partitioned data; protocols such as 2PC and 3PC may be used, but implementations vary.

4. Analytics

Analytics covers SQL parsing, logical planning, and physical execution planning, including join ordering, predicate push‑down, and operator implementation choices. Optimizations rely on heuristics, cost models, or adaptive feedback.

Tuple‑at‑a‑time (iterator model).

Full materialization (batch processing).

Vectorized execution (columnar, cache‑friendly loops).

Columnar storage, vectorized engines, and MPP architectures are key technologies for modern analytical databases.

5. Multi‑Core

Scaling on many cores faces Amdahl’s law; reducing contention via lock‑free algorithms, careful scheduling, and balanced task partitioning is essential. Multi‑core scaling introduces challenges similar to distributed systems, requiring efficient inter‑core communication and memory hierarchy awareness.

6. Computation

The execution engine’s roadmap is defined, but a baseline implementation is still pending.

7. Compilation

Compilation techniques permeate databases: vectorized execution can be further optimized with JIT compilation, case‑by‑case performance improvements often require deep architectural research, and DSLs for stream‑batch processing rely on compiled SQL engines and UDF extensions.