Debugging Redis Lettuce Timeout Issues in a Spring Cloud Backend

As a backend developer familiar with Redis, the author encountered frequent Redis timeout errors after upgrading Docker images in an iQIYI overseas backend project. The issue only appeared with the custom cache framework that uses Lettuce, not with Spring's RedisTemplate.

The project uses a Spring Cloud stack and a Redis cluster for caching program details, episode lists, and playback authentication. Two access methods exist: direct RedisTemplate calls and indirect calls through a self‑developed cache framework that adds features like second‑level caching and hot‑key statistics.

After the image upgrade, the application started normally but soon produced many Redis timeout errors. Tests showed that RedisTemplate accesses remained normal, while the cache framework timed out when connecting to the Redis cluster, though it recovered after a while.

Investigation revealed that the cache framework directly uses Lettuce (the low‑level Redis client) instead of Spring. A minimal reproducible case was created, simulating the cache warm‑up process where a new node receives HOTKEY messages, looks up values from Redis, and stores them locally.

Monitoring the TCP connections showed that one of the six connections to the Redis cluster had a constantly non‑empty receive buffer, indicating that data arrived but was not consumed. This pointed to a blocked Netty EventLoop thread.

Further analysis of Lettuce’s architecture showed that it relies on Netty’s NIO model with a limited number of EventLoop threads handling multiple connections. The cache framework creates several connections (main, replica, Pub/Sub, and subscription) that are all registered to the same EventLoop group.

Using Arthas, it was discovered that the Pub/Sub listener thread (epollEventLoop‑9‑3) was blocked by a synchronous future.get() call inside the hot‑key processing callback, preventing the EventLoop from handling other I/O events and causing buffer buildup.

The root cause is the blocking operation in the Pub/Sub callback, which monopolizes the EventLoop thread. The solution is to avoid blocking in Netty threads: either increase the number of I/O threads so Pub/Sub and other connections use different EventLoops, or off‑load the processing to separate worker threads (as Spring Data Redis does).

Two concrete fixes were demonstrated: configuring Lettuce’s ClientResources to increase I/O threads, or setting the Netty property io.netty.eventLoopThreads . After applying these changes, the timeout issue disappeared.

Additional insights include why lower‑version Docker images did not show the problem (they inadvertently created more EventLoop threads due to JDK version differences) and why Spring‑managed Redis connections were unaffected (they use a separate EventLoop group).

The article concludes with a summary of the findings, emphasizing the importance of understanding EventLoop behavior, avoiding blocking operations in Netty callbacks, and properly configuring I/O thread counts for stable Redis access.

References: Lettuce documentation, Spring Data Redis Pub/Sub guide, Netty learning resources, JetCache Redis integration, Arthas thread analysis, and Java Docker CPU limit articles.