Building a Stable High‑Traffic Microservice System with Open‑Source Components and Consul‑Based Service Discovery

This article summarizes a talk by Tencent expert engineer Liu Zhixin on how to build a stable, high‑traffic microservice system using open‑source components. It introduces the architecture of a microservice platform, the challenges of data‑plane and control‑plane, and the need for reliable service registration and discovery.

The core of the solution is a Consul‑based service discovery system. The native Consul open‑source version lacks enterprise features such as multi‑tenant support. To address this, a Consul‑access layer is added in front of the Consul server, providing multi‑tenant capabilities, token‑based authentication, and request translation to KV operations.

Multi‑tenant is achieved by storing service instances under paths like /tenant/service/ServiceA/instance-id . The access layer intercepts registration, heartbeat, and discovery requests, aggregates them, and reduces the number of long‑living connections to the Consul server, greatly improving scalability.

Performance optimizations include aggregating watch requests, extending watch timeout from 2 s to 55 s, and reducing CPU usage on the Consul server. Horizontal scaling of the stateless access layer further alleviates pressure on the Consul leader.

High‑availability measures are discussed: handling registration failures, heartbeat loss, and empty instance lists; switching from periodic polling to watch mechanisms; adding local caches and zero‑instance protection; configuring timeouts and retries; and using circuit‑breaker (Hystrix/Resilience4j) and fallback strategies.

The article also covers graceful shutdown and rolling updates in Kubernetes using pre‑stop hooks, readiness probes, and coordination with Consul to avoid traffic spikes during instance deregistration.

Overall, the talk provides practical guidance on enhancing service discovery stability, performance, and availability in cloud‑native microservice environments.