Understanding Distributed Systems: Zookeeper, 2PC/3PC, Consensus Algorithms, CAP and BASE Theories

Preface

One year ago I wrote a summary about Zookeeper; now I revisit it and clarify the background and concepts.

1. Relationship Between Distributed Systems and Zookeeper

1.1 Centralized Services

Early deployments were centralized: a single machine handled all services, typically a web app on Tomcat (port 8080) and a database on port 3306. Scaling could be vertical (more resources) or horizontal (more machines), but single‑point failures and high costs made pure centralization impractical.

1.2 "IOE" Migration

Alibaba promoted moving away from IBM mainframes, Oracle databases, and EMC storage to cheaper, distributed solutions, sparking the rise of distributed architectures in China.

1.3 Distributed Services

A distributed system consists of hardware or software components spread across different networked computers that communicate only via message passing, appearing to the user as a single service.

Key characteristics (five points):

Distribution – components are placed in different locations.

Peer‑to‑peer – all nodes perform the same role; replicas exist.

Concurrency – simultaneous operations can cause inconsistency.

Global clock – ordering of events across nodes is complex.

Faults – node crashes, network partitions, etc.

1.4 Common Problems in Distributed Scenarios

Communication anomalies – network issues causing data inconsistency.

Network isolation – sub‑networks work but the whole system does not.

Node crashes.

Three‑state outcomes – success, failure, timeout, leading to lost or ambiguous responses.

Data loss – mitigated by replicas or state recovery.

Principle: any exception considered during design must be assumed to occur in production.

1.5 Performance Metrics for Distributed Systems

Performance – throughput, latency, concurrency (often measured by QPS).

Availability – e.g., “five nines” means only 5 minutes of downtime per year.

Scalability – ability to improve performance by adding nodes.

Consistency – replica management.

High availability often requires more replicas, which can weaken consistency; trade‑offs must be evaluated per business scenario.

1.6 Consistency Variants

Strong consistency – a read after a write always sees the latest value (e.g., Paxos, Quorum, ZAB).

Weak consistency – no guarantee of immediate visibility; only eventual convergence.

Final consistency is a special weak form where the system strives to make data consistent as quickly as possible, though exact timing is undefined.

2. Distributed Transactions

A transaction guarantees atomicity across multiple operations; in distributed systems a coordinator and participants are needed. The classic protocols are 2‑Phase Commit (2PC) and 3‑Phase Commit (3PC).

2.1 What Is 2PC?

Participants execute the transaction but do not commit until the coordinator instructs them. If all participants report success, the coordinator sends a commit; otherwise it sends a rollback.

2PC Phase 1 – Execute

The coordinator asks each participant to execute and log the operation without committing.

2PC Phase 2 – Commit

After receiving successful acknowledgments, the coordinator issues a commit; any missing acknowledgment leads to a rollback.

Drawbacks of 2PC

Performance – synchronous blocking while waiting for all nodes.

Single‑point failure – coordinator crash stalls the transaction.

Data inconsistency – some participants may miss the final commit.

No fault tolerance – any participant failure forces a rollback.

2.2 What Is 3PC?

3PC adds an extra “pre‑commit” stage and a timeout mechanism: if participants do not receive a final decision within a timeout, they locally commit, reducing coordinator‑centric single‑point failure.

3. Distributed Consensus Algorithms

3.1 Paxos

Paxos, introduced by Leslie Lamport, is a message‑based, highly fault‑tolerant algorithm that ensures consistency despite crashes, delays, or lost messages. Zookeeper’s ZAB protocol implements Paxos under the hood.

3.2 Roles in Paxos and Their Zookeeper Mapping

Proposer → leader, Acceptor → follower, Learner → observer.

3.3 Raft

Raft is a consensus protocol that elects a leader among three possible states: follower, candidate, and leader. Nodes start as followers, become candidates when they cannot find a leader, and are elected leader after receiving votes from a majority.

3.4 ZAB Protocol

ZAB (Zookeeper Atomic Broadcast) provides crash recovery and message broadcasting, ensuring that transactions committed on the leader are eventually committed on all servers.

3.5 Quorum NWR Mechanism

Quorum defines N (total replicas), W (writes that must succeed), and R (reads needed for latest data). The rule W + R > N guarantees that a read will see the most recent write.

W = 1, R = N   // Write Once, Read All

R = 1, W = N   // Read Only, Write All

W = Q, R = Q where Q = N/2 + 1   // Balanced read/write

3.6 CAP Theorem

In a distributed system you can only satisfy two of Consistency, Availability, and Partition tolerance. Since Partition tolerance is mandatory, a trade‑off between Consistency and Availability must be made.

3.7 BASE Theory

BASE (Basically Available, Soft state, Eventually consistent) is a pragmatic response to CAP, accepting weaker consistency for higher availability.

Finally

We reviewed centralized vs. distributed architectures, consistency challenges, 2PC/3PC drawbacks, and how algorithms like Paxos, Raft, and ZAB address them. Quorum NWR, CAP, and BASE illustrate the inevitable trade‑offs between consistency and availability in real‑world systems.

Most production systems adopt a BASE approach: basic availability with eventual consistency.