How Google’s Chubby Inspired Zookeeper’s Distributed Coordination and Locking

Chubby

Chubby is a distributed lock service described in Google’s 2006 paper "The Chubby lock service for loosely‑coupled distributed systems". It provides coarse‑grained locking and low‑volume reliable storage for loosely coupled systems, and is used internally by GFS and Bigtable for leader election and configuration distribution.

We describe our experiences with the Chubby lock service, which is intended to provide coarse‑grained locking as well as reliable (though low‑volume) storage for a loosely‑coupled distributed system.

Chubby’s architecture consists of a server component that offers read/write APIs and a client SDK. Each Chubby cell contains a group of servers that run a consensus algorithm to elect a master. Only the master serves client reads and writes; other nodes hold replicas that synchronize state from the master. If the master fails, a new master is elected after the lease expires.

Zookeeper

Zookeeper is a widely known open‑source distributed coordination service that offers features such as publish‑subscribe, naming, distributed locks, and more. While often described as an open‑source implementation of Chubby, the two differ fundamentally: Chubby is a dedicated lock service, whereas Zookeeper provides a richer coordination API that clients use to build higher‑level primitives.

In this paper, we describe ZooKeeper, a service for coordinating processes of distributed applications.

Zookeeper’s cluster has a leader and followers. All nodes can serve reads, but only the leader handles writes. Clients can connect to any node; writes are forwarded to the leader, which then broadcasts the transaction to followers using the Zab (ZooKeeper Atomic Broadcast) protocol.

Zab Protocol

Zab ensures high availability by broadcasting transaction proposals and committing them once a majority of nodes acknowledge. It uses a 64‑bit transaction identifier (ZXID) where the high 32 bits represent the epoch (leader term) and the low 32 bits are a monotonically increasing counter.

Zab vs. Paxos

Zab shares many similarities with Paxos—proposals are sent by the leader, a majority of followers must acknowledge before a proposal is committed, and each proposal carries an epoch value similar to Paxos’s ballot. However, Zab is optimized for primary‑backup systems, while Paxos targets generic state‑machine replication.

Implementation Details

Zookeeper stores data in a hierarchical namespace of znodes. Each znode can be one of four types: PERSISTENT, PERSISTENT_SEQUENTIAL, EPHEMERAL, or EPHEMERAL_SEQUENTIAL. Ephemeral nodes disappear when the client session ends, while sequential nodes receive an auto‑incremented suffix.

Clients receive change notifications via watches. When a client calls getData (or similar) with a Watcher, the server pushes events to the watcher when the node changes.

public byte[] getData(final String path, Watcher watcher, Stat stat)

The watch manager triggers the callback:

Set<Watcher> triggerWatch(String path, EventType type, Set<Watcher> suppress) {<br/>    WatchedEvent e = new WatchedEvent(type, KeeperState.SyncConnected, path);<br/>    Set<Watcher> watchers;<br/>    synchronized (this) {<br/>        watchers = watchTable.remove(path);<br/>    }<br/>    for (Watcher w : watchers) {<br/>        w.process(e);<br/>    }<br/>    return watchers;<br/>}

Sessions are central to Zookeeper. Each client connection creates a session with a unique ID, timeout, and state (CONNECTING, CONNECTED, RECONNECTING, etc.). The SessionTracker interface manages session lifecycle.

public interface SessionTracker {<br/>    interface Session {<br/>        long getSessionId();<br/>        int getTimeout();<br/>        boolean isClosing();<br/>    }<br/>    interface SessionExpirer {<br/>        void expire(Session session);<br/>        long getServerId();<br/>    }<br/>    long createSession(int sessionTimeout);<br/>    boolean trackSession(long id, int to);<br/>    boolean commitSession(long id, int to);<br/>    boolean touchSession(long sessionId, int sessionTimeout);<br/>    void setSessionClosing(long sessionId);<br/>    void shutdown();<br/>    void removeSession(long sessionId);<br/>}

Applications

Using Zookeeper, developers can implement:

Publish‑subscribe for dynamic configuration.

Naming services with unique sequential nodes.

Distributed transaction coordination (via ZAB and multi‑operation APIs).

Distributed locks using ephemeral sequential znodes.

Publish‑Subscribe Example

ZooKeeper zk = new ZooKeeper("localhost", 3000, null);<br/>zk.getData("/config", new Watcher() {<br/>    public void process(WatchedEvent e) {<br/>        System.out.println(e.toString());<br/>    }<br/>}, null);<br/>zk.setData("/config", "draven".getBytes(), 0);

Naming Service Example

ZooKeeper zk = new ZooKeeper("localhost", 3000, null);<br/>zk.create("/metrics", new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT_SEQUENTIAL);<br/>zk.create("/metrics", new byte[0], ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT_SEQUENTIAL);<br/>List<String> children = zk.getChildren("/", null);<br/>System.out.println(children); // [metrics0000000001, metrics0000000002]

Distributed Lock Example

ZooKeeper zk = new ZooKeeper("localhost", 3000, null);<br/>String lockPath = zk.create("/resource/lock-", null, ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL_SEQUENTIAL);<br/>List<String> locks = zk.getChildren("/resource", false, null);<br/>Collections.sort(locks);<br/>if (locks.get(0).equals(lockPath.substring("/resource/".length()))) {<br/>    System.out.println("Acquire Lock");<br/>    zk.delete(lockPath, 0);<br/>} else {<br/>    // watch predecessor node<br/>}

These patterns illustrate how Zookeeper provides a reliable coordination layer for distributed systems, complementing other services such as Redis or etcd for lock implementation.

Summary

The article introduced Google’s Chubby lock service and compared it with Apache Zookeeper, highlighting Zookeeper’s broader coordination capabilities, its Zab protocol, session management, and practical usage scenarios including configuration publishing, naming, transaction coordination, and distributed locking.

References

Distributed Coordination with Zookeeper (https://www.igvita.com/2010/04/30/distributed-coordination-with-zookeeper/)

ZooKeeper: Wait‑free coordination for Internet‑scale systems (https://www.usenix.org/legacy/event/usenix10/tech/full_papers/Hunt.pdf)

The Chubby lock service for loosely‑coupled distributed systems (https://ai.google/research/pubs/pub27897)

Zookeeper vs Chubby (http://catkang.github.io/2017/10/10/zookeeper-vs-chubby.html)

Zab vs. Paxos (https://cwiki.apache.org/confluence/display/ZOOKEEPER/Zab+vs.+Paxos)

Zab: High‑performance broadcast for primary‑backup systems

ZooKeeper Recipes and Solutions (https://zookeeper.apache.org/doc/r3.4.4/recipes.html)