Understanding ZooKeeper: Purpose, Features, and Design Goals

Goal

ZooKeeper is widely used, and a basic question arises: what is ZooKeeper used for, and why was it created?

ZooKeeper simplifies distributed application development by hiding low‑level coordination details.

It exposes a simple API that supports distributed application development.

It is a high‑performance, highly available, and reliable distributed cluster.

In short, ZooKeeper solves coordination problems in distributed applications.

Why ZooKeeper Exists

When multiple processes need to cooperate, business logic becomes tangled with complex coordination code.

The multi‑process coordination logic has two characteristics:

Complex processing

Reusable logic

Therefore, the common coordination problems are extracted as infrastructure so that developers can focus on business logic.

ZooKeeper is one such coordination service.

ZooKeeper Characteristics

ZooKeeper has a few simple characteristics:

Its API is inspired by a file‑system API and provides a simple interface.

It runs on dedicated servers, separated from business logic, ensuring high fault‑tolerance and scalability.

ZooKeeper is a storage facility, but it focuses on storing coordination metadata, not application data (which should be stored elsewhere, e.g., HDFS).

Application data and metadata have different consistency and durability requirements; they should be treated and stored separately.

ZooKeeper Mission

The core problems ZooKeeper aims to solve are:

Unified naming service

Distributed locks

Process crash detection

Leader election

Configuration management (propagating config changes to clients)

Multi‑process coordination can be classified into two types:

Collaboration: multiple processes work together on a task (e.g., master‑slave task assignment).

Competition: only one process may act at a time (e.g., leader election after a master fails).

Collaboration can be:

Intra‑node (same physical machine) using primitives like pipes, shared memory, message queues, semaphores.

Inter‑node (different machines) – the scenario ZooKeeper addresses.

Cross‑network coordination faces three common issues:

Message latency (out‑of‑order delivery)

Processor scheduling delays

Clock skew across machines

ZooKeeper is designed to hide these three problems, making them transparent to the application layer.

ZooKeeper Features

Fundamental Problem Solved

Distributed consistency challenges:

Message delay

Message loss

Node crashes

ZooKeeper uses proposal voting (based on Paxos) and leader election to achieve consensus and ensure that a majority of nodes see the same state.

Positioning

Distributed coordination service

High performance, high availability, high reliability

Allows developers to focus on business logic without dealing with low‑level coordination details

Instead of exposing low‑level primitives, ZooKeeper provides a set of API calls similar to a file system, enabling applications to build their own primitives.

Consistency Guarantees

Sequential consistency: client requests are processed in the order they were issued.

Atomicity: a transaction is applied to all nodes or none.

Single view: all clients see the same data (effectively eventual consistency).

Durability: once a transaction succeeds, its state is permanent.

Timeliness: clients may not see the latest data immediately, but ZooKeeper guarantees eventual consistency.

Design Goals

Goal 1: High Performance (Simple Data Model)

Tree‑structured data nodes

All data stored in memory

Followers and observers handle read‑only requests

Goal 2: High Availability (Cluster Construction)

Service remains operational as long as a majority of machines are alive

Automatic leader election

Goal 3: Sequential Consistency (Ordered Transactions)

All transaction requests are forwarded to the leader

Each transaction receives a globally unique, monotonically increasing zxid (64‑bit: epoch + counter)

Goal 4: Eventual Consistency

Proposal voting ensures reliable transaction commits

After a commit, a majority of nodes will see the new state

Before ZooKeeper

Prior to ZooKeeper, distributed systems typically used either a distributed lock manager or a distributed database to achieve coordination.

ZooKeeper focuses specifically on process coordination and does not provide built‑in lock or generic storage interfaces (though they can be built on top of its API).

Typical application server needs include:

Master‑slave leader election

Process crash detection

Distributed locks

ZooKeeper supplies the basic APIs to address these needs.

Unsuitable Scenarios

Massive data storage – ZooKeeper is meant for metadata, not bulk application data.

Glossary

References

ZooKeeper – Distributed Process Coordination, Chapter 1 Introduction

From Paxos to ZooKeeper: Distributed Consistency Principles and Practice, Chapter 4 Introduction to ZooKeeper