Design of a Cloud-Native High-Availability Service Naming Center

点击关注公众号👆，探索更多Shopee技术实践

目录
1. 背景
2. 名词介绍
3. 架构介绍
4. 可用性设计
5. 一致性设计
6. 正确性保证
7. 升级过程
8. 未来探讨

In a cloud‑native architecture, most services are deployed as containers and rely on a service registry for inter‑service communication. High availability of the registry is therefore critical.

Existing solutions fall into two categories: (1) general KV stores such as ZooKeeper, etcd, Consul that provide strong consistency; (2) integrated service‑registry suites like Spring Cloud Eureka and Tencent Polaris that include traffic‑control features.

While the former offers linear consistency, the latter simplifies integration. ShopeePay initially used etcd but encountered issues: etcd sacrifices availability for consistency, is hard to scale horizontally, and its generic KV features add unnecessary complexity.

To address these problems, we built a cloud‑native, high‑availability naming service that works with the sRPC framework.

Terminology

Naming Service : the server side of the service registry.

Naming Client : the client SDK used by services.

sRPC : ShopeePay microservice framework that incorporates the Naming Client.

Instance : a service process, typically a pod.

Replica : a Naming Service node.

Replicator : module that synchronizes data between Naming Service nodes.

Instance Table : data structure storing Instance information.

Route Table : IP/port data of services without lease information.

Architecture

The system consists of three components: Naming Client, Naming Service, and Coordinator.

Naming Client: SDK that registers and discovers services.

Naming Service: independent RPC service that stores routing information.

Coordinator: control‑plane component that monitors status, data consistency, and load balancing.

Availability Design

The service sacrifices strong consistency for eventual consistency to achieve high availability.

Key design points:

In‑memory storage instead of persistent storage.

Peer‑to‑peer node design (no leader).

Avoidance of single points of failure.

Reduced external dependencies.

In‑Memory Storage

Both Naming Service and Naming Client keep routing data in memory, allowing rapid recovery after crashes and seamless container scheduling.

Peer‑to‑Peer Nodes

All Naming Service nodes are equal; any client can connect to any node. New nodes join by contacting any existing node, and the Coordinator can balance load by directing clients to less‑loaded nodes.

Avoiding Single Points

Both client and server sides are designed to eliminate single‑point failures.

Discovery of Naming Service

Clients locate services via DNS registration, a fallback configuration service, or cached IP lists.

Client Cache

Naming Client maintains an LRU HashMap cache of route information, refreshed periodically and updated via service change events.

Reducing Dependencies

Configuration is delivered through the Naming Service itself via gRPC streams, avoiding extra configuration‑center dependencies.

Consistency Design

Multiple replicas synchronize using a gossip‑like protocol with heartbeat broadcasting.

Key Mechanisms

Heartbeat doubles as registration.

Synchronous heartbeat propagation between replicas.

Self‑protection mechanism to prevent premature removal of instances during network partitions.

Algorithm Overview

New Naming Service registers to DNS in init state, not discoverable.

It fetches a full Instance Table snapshot from a random existing node.

It registers its own instance to all nodes, which then propagate the data.

It enters service state, accepting client connections and exchanging heartbeats.

Heartbeat payload is an array of entries containing instance ID, service name, IP, port, timestamp, and optional data.

Self‑Protection Trigger

If a node loses heartbeats from more than 1/N of the known Naming Service nodes (where N is the total number of nodes) and at least one node’s heartbeat is missing, it enters self‑protection mode, refraining from removing stale instances while still accepting new registrations.

Correctness Guarantees

Chaos testing with multiple nodes, random restarts, and kill‑operations validates data consistency. The Coordinator performs periodic reconciliation:

Minute‑level quick checks using summary metrics.

Hourly full‑snapshot comparisons.

Cross‑check with etcd during gray‑release.

Instance Table uses segmented locks and iterator‑based snapshots to avoid blocking lease updates.

Upgrade Process

During migration from etcd to the new Naming Service, services perform dual writes to both registries. Once the new registry is stable, the etcd dependency can be removed.

Future Work

Sharding Storage

Introduce partitioned storage where each Naming Service node holds only a subset of instances, with request forwarding for out‑of‑range data.

Load‑Aware Service Discovery

Expose service load metrics to the sRPC framework to enable smarter downstream node selection.

Authors: Xin (ShopeePay team) and Yinhang (Engineering Infrastructure team).