Implementing a Custom Protocol with Netty: Design, Encoding/Decoding, and Heartbeat

The article begins by comparing the widely used HTTP protocol with a custom TCP protocol, highlighting three main drawbacks of HTTP: low byte utilization due to headers and cookies, the overhead of TCP short‑connection handshakes for high‑concurrency scenarios, and the inability to satisfy business‑specific communication requirements.

It then specifies the custom protocol structure, which consists of a fixed‑length header (magic number, main/sub/modify version, sessionId, message type, attachment count) and a variable‑length body. An illustrative diagram (image) shows the layout of these fields.

Implementation details start with the Message class that models the protocol fields and the MessageTypeEnum defining REQUEST, RESPONSE, PING, PONG, and EMPTY types. The MessageEncoder (extending MessageToByteEncoder ) converts a Message instance into a byte stream, while the MessageDecoder (extending ByteToMessageDecoder ) performs the reverse operation.

A singleton MessageResolverFactory holds a list of Resolver implementations. Specific resolvers— RequestMessageResolver , ResponseMessageResolver , PingMessageResolver , and PongMessageResolver —determine how each message type is processed and generate appropriate responses or empty messages.

The server bootstrap creates boss and worker groups, configures the pipeline with LengthFieldBasedFrameDecoder , LengthFieldPrepender , the custom encoder/decoder, and the ServerMessageHandler . The client bootstrap adds an IdleStateHandler for heartbeat detection, the same encoder/decoder, and the ClientMessageHandler , which sends random request messages and triggers Ping messages when the channel is idle.

Heartbeat logic is implemented in ClientMessageHandler.userEventTriggered : if no inbound data is read for a configured interval, a Ping message is sent; the server replies with a Pong, and the client logs the receipt.

Running the program produces console logs that show the exchange of Ping/Pong, request/response messages, and attachment data, confirming that the custom protocol, encoding/decoding, and heartbeat mechanisms work as intended.

In summary, the article demonstrates the advantages of a tailor‑made protocol over HTTP, provides a full specification, and delivers a complete Netty‑based Java implementation that can be used as a reference for building high‑performance, business‑specific communication layers.