Design and Optimization of Massive Push Services Using Netty

Background

Push services have become essential in mobile apps and IoT devices, but they face massive connection counts, unstable wireless networks, and high resource consumption.

Topic Source

Can Netty be used to build a push server?

How many clients can a Netty‑based push server support?

Technical problems encountered when developing push services with Netty.

The author collected these questions from many developers and aims to provide a case‑based analysis and design summary.

Push Service Overview

Push services increase user activity and retention; with IoT, virtually every smart device becomes a push client, leading to massive scale.

Characteristics of Mobile Push Services

Unstable wireless networks cause frequent disconnections.

Massive long‑lived connections consume significant resources.

Android long connections are maintained per app, generating large heartbeat traffic.

Message loss, duplicate pushes, latency, and expiration are common.

Spam and lack of unified governance degrade service quality.

Some vendors (e.g., JD Cloud) offer multi‑app single‑service connection models and heartbeat‑based power saving.

Real‑World Smart‑Home Case

Problem Description

An MQTT middleware kept 100k users online with 20k concurrent requests. After running for a while, memory leakage was suspected to be a Netty bug.

Server: 16 GB RAM, 8‑core CPU.

Boss thread pool = 1, worker thread pool = 6 (later increased to 11) – problem persisted.

Netty version 4.0.8.Final.

Diagnosis

Heap dump revealed a 9076 % increase in ScheduledFutureTask instances (≈1.1 M). The cause was an IdleStateHandler with a 15‑minute idle timeout, creating a scheduled task per connection.

Each task held references to business objects, preventing GC. Reducing the timeout to 45 seconds allowed normal memory reclamation and solved the issue.

Problem Summary

With only 100 connections, long‑period tasks are harmless, but at 100 k connections they amplify minor issues into major problems.

The following sections outline Netty‑specific design points for supporting millions of clients.

Netty Massive Push Service Design Points

Max File Descriptor Adjustment

Linux’s default limit (1024) is insufficient for millions of connections. Use ulimit -a to view limits and edit /etc/security/limits.conf :

*    soft    nofile    1000000
*    hard    nofile    1000000

After adjustment, restart the session and verify with ulimit -a .

Beware of CLOSE_WAIT

Unstable networks cause many client resets. If the server does not close the socket promptly, connections linger in CLOSE_WAIT, consuming file descriptors and memory, eventually leading to “Too many open files”.

Typical causes:

Bug in Netty or business code that fails to close the socket after receiving FIN.

I/O thread blockage (e.g., long‑running tasks, high‑load logging) prevents timely socket closure.

Reasonable Heartbeat Period

Mobile networks often drop idle connections after a few minutes. A heartbeat interval of around 180 seconds (e.g., 300 s for WeChat) balances connection stability and signaling overhead.

Example of adding IdleStateHandler in Netty:

public void initChannel(Channel channel) {
    channel.pipeline().addLast("idleStateHandler",
        new IdleStateHandler(0, 0, 180));
    channel.pipeline().addLast("myHandler", new MyHandler());
}

public class MyHandler extends ChannelHandlerAdapter {
    @Override
    public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
        if (evt instanceof IdleStateEvent) {
            // handle heartbeat
        }
    }
}

Buffer Size Configuration

Each long‑lived connection holds receive and send buffers. Using fixed‑size ByteBuffer can waste memory; instead, leverage Netty’s dynamic ByteBuf and allocators.

Two allocators:

FixedRecvByteBufAllocator – fixed size, can expand if needed.

AdaptiveRecvByteBufAllocator – adjusts capacity based on recent traffic.

Example configuration:

Bootstrap b = new Bootstrap();
 b.group(group)
  .channel(NioSocketChannel.class)
  .option(ChannelOption.TCP_NODELAY, true)
  .option(ChannelOption.RCVBUF_ALLOCATOR, AdaptiveRecvByteBufAllocator.DEFAULT);

Memory Pool

Using a pooled ByteBuf (e.g., PooledByteBufAllocator ) reduces allocation and GC pressure. Netty’s pool works like a Java‑based jemalloc.

Enable it when building the client/server:

Bootstrap b = new Bootstrap();
 b.group(group)
  .channel(NioSocketChannel.class)
  .option(ChannelOption.TCP_NODELAY, true)
  .option(ChannelOption.ALLOCATOR, PooledByteBufAllocator.DEFAULT);

Remember to release buffers with ReferenceCountUtil.release(msg) to avoid leaks.

Avoid the “Log Hidden Killer”

Synchronous logging (e.g., Log4j without a properly sized async queue) can block I/O threads when disk I/O is high, leading to connection stalls and CLOSE_WAIT buildup.

TCP Parameter Optimization

Adjust SO_SNDBUF and SO_RCVBUF (≈32 KB is a good start) and enable Receive Packet Steering (RPS) on Linux ≥ 2.6.35 to distribute soft‑interrupts across CPUs, improving throughput by >20 %.

JVM Parameters

Set appropriate -Xmx based on memory model and tune GC (young/old generation sizes, collector choice) to minimize Full GC pauses.

---

For further reading, see the recommended articles linked at the end of the original source.