Design and Implementation of a High‑Availability Twemproxy‑Based Redis/Memcached PaaS at Meitu

Abstract

Meitu started building a Redis/Memcached resource PaaS in late 2017. To make scaling invisible to services, the team introduced twemproxy as a gateway in November 2017. The upstream open‑source version could not fully meet Meitu's needs due to its single‑threaded model, lack of online configuration reload, no Redis master‑slave support, and missing latency metrics, so the team applied several modifications, adding multi‑process capability, online config reload, and latency monitoring.

Why Choose twemproxy

twemproxy, an open‑source Redis/Memcached proxy from Twitter, reduces backend connection counts and enables horizontal scaling. It supports various hash sharding algorithms and automatic removal of failed nodes. Compared with alternatives like Codis, which is closer to Redis Cluster, twemproxy better fits Meitu's need for a protocol‑level proxy for both Redis and Memcached, leading to its selection.

twemproxy Implementation

twemproxy parses client requests and forwards them to backend cache resources, then returns the responses to the client.

The core implementation revolves around three connection objects:

proxy connection – listens for incoming client connections and creates a corresponding client connection.

client connection – handles client read/write, parses requests, selects a server based on the key and hash rule, and forwards the request.

server connection – forwards the request to the cache server, receives and parses the response, and sends it back to the client connection.

The data flow among these connections is illustrated below:

(The client connection does not have an imsgq because the request can be directly placed into the server's imsgq after parsing.)

The user establishes a connection via proxy connection, creating a client connection.

The client connection reads the request, selects a server according to the hash rule, and stores the request in the server's imsgq.

The server connection sends the imsgq request to the remote resource, moves the message to omsgq after writing to the TCP buffer, and later retrieves the response from omsgq.

The response is placed into the client connection's omsgq and sent back to the user.

All request and response data are kept in memory buffers; the internal flow between client and server connections is essentially pointer copying (Zero Copy), allowing the single‑threaded model to achieve up to 100k QPS for small packets.

Problems with the Upstream Version

Single‑threaded model cannot utilize multiple CPU cores, limiting performance.

Configuration cannot be reloaded online; the original Twitter version supports it, but the open‑source branch does not.

Redis master‑slave mode is not supported, which is required for some of Meitu's use cases.

Latency metrics are absent.

Multi‑Process Version

To address the above issues, Meitu added multi‑process support and online configuration reload. The process model resembles Nginx, with a master process managing worker processes.

The master does not handle client requests; it only spawns and monitors workers. If a worker exits unexpectedly, the master restarts it. The master also handles several signals:

SIGHUP – reload configuration.

SIGTTIN – increase log level.

SIGTTOU – decrease log level.

SIGUSR1 – reopen log file.

SIGTERM – graceful shutdown.

SIGINT – forceful shutdown.

Additional global configuration options are introduced, such as worker_shutdown_timeout , which defines how long an old worker may stay alive after receiving a termination signal, enabling seamless configuration and worker count changes.

Reuse Port

Before reuse‑port, two common approaches existed for accepting connections: a single thread handling all accepts (becoming a bottleneck) or all threads/processes sharing the same listening socket (causing wake‑up imbalance and thundering‑herd). Reuse‑port allows multiple sockets to listen on the same port without uneven connection distribution. Implementation simply sets the SO_REUSEPORT flag on the listening sockets.

During reload, old listening sockets must not be closed abruptly to avoid losing connections in the TCP backlog. Meitu solves this by creating listening sockets in the master process; workers inherit them after fork, allowing the master to transfer listening sockets to new workers during reload.

Relevant code can be found in nc_process.c#L172 , which ensures backlog data is preserved when scaling workers up or down.

Redis Master‑Slave Mode

Upstream twemproxy does not support Redis master‑slave because it treats Redis/Memcached as pure caches. Meitu added simple support: if a server name is "master", it is treated as the primary instance, and only one master is allowed per pool.

Metrics

twemproxy originally lacked latency metrics, making troubleshooting difficult. Two latency metrics were added:

request latency – total time from client request to response, including proxy and server processing.

server latency – time spent inside the Redis/Memcached server only.

These metrics are recorded in buckets (e.g., <1 ms, <10 ms) and help pinpoint whether slow requests stem from the proxy, the backend server, or the client.

Remaining Issues

When reducing worker count, the TCP backlog of terminated workers may be lost, causing connection timeouts.

Unix sockets lack a reuse‑port‑like mechanism, so they remain single‑process.

Binary Memcached protocol is not supported; a PR exists but is not merged.

Client‑side max‑connection configuration exists but is not effective yet.

Some commands (especially those without keys or blocking commands) are not supported.

Configuration inconsistencies between old and new workers during reload may produce dirty data.

Performance Benchmark

Benchmarks were conducted with long‑lived small‑packet connections to verify the multi‑process version meets expectations. Performance scales linearly with CPU cores until the network interface becomes the bottleneck.

Test environment:

CentOS 6.6

Intel E5‑2660, 32 logical cores

64 GB RAM

Two 1 Gbps NICs bonded as bond0

A single worker achieves around 100k QPS, comparable to the original twemproxy. Adding workers increases throughput proportionally until the bond0 NIC becomes the limiting factor.

Conclusion

The multi‑process twemproxy implementation is relatively simple, but the development process uncovered many subtle issues (e.g., mbuf growth without shrinkage). Ongoing optimizations continue, and the project is maintained as a single repository on GitHub.

Code repository: https://github.com/meitu/twemproxy

Other open‑source projects from the team include Golang and PHP Kafka consumer groups and an Ethereum‑based DPoS implementation.

We will keep releasing more open‑source work; stay tuned!

Thanks to contributors: @ruoshan, @karelrooted, @Huang‑lin, @git‑hulk

References

1. twitter/twemproxy https://github.com/twitter/twemproxy

2. meitu/twemproxy https://github.com/meitu/twemproxy

3. twemproxy binary memcached https://github.com/twitter/twemproxy/tree/binary_memcache

4. reuseport: TCP/IPv4 implementation https://lwn.net/Articles/542718/

5. The SO_REUSEPORT socket option https://lwn.net/Articles/542629/