Analyzing and Resolving TCP CLOSE_WAIT Issues in Java HTTP Client with Apache HttpClient

Background and Problem Description

The service experienced a sudden increase in CLOSE_WAIT TCP connections, causing resource exhaustion and alerts. Initial investigation showed many connections stuck in the CLOSE_WAIT state after the server sent a FIN packet.

TCP State Analysis

Using tcpdump the author reconstructed the three‑way handshake, data transfer, and four‑way termination sequence, confirming that the server (nginx) correctly sent FIN and the client entered CLOSE_WAIT because it never called socket.close() or closed the input stream.

Root Cause in Java Code

Further inspection revealed that each HTTP request created a new HttpClient instance:

HttpClient client = new HttpClient(new MultiThreadedHttpConnectionManager());

Because the HttpClient was instantiated inside the business method, the underlying HttpConnection was returned to the pool but never explicitly closed. The connection remained idle until the JVM garbage collector invoked finalize() , which called close() on the socket, finally releasing the CLOSE_WAIT state.

Impact of HTTP Keep‑Alive

The remote nginx server used keepalive_timeout 30 , so after 30 seconds it closed idle connections with a FIN . The client’s connection pool kept the socket open, leading to the observed CLOSE_WAIT buildup.

Apache HttpClient Connection Pool Mechanics

The MultiThreadedHttpConnectionManager maintains per‑host pools with defaults http.connection-manager.max-per-host=2 and http.connection-manager.max-total=20 . Connections are obtained via getConnectionWithTimeout() and returned with releaseConnection() . If alwaysClose is false (default), releaseConnection() only returns the socket to the pool; the socket is closed only when the pool discards it or when shutdown() is called.

Common Pitfalls and Fixes

Never create a new HttpClient per request; reuse a singleton or a bean‑scoped instance.

When using SimpleHttpConnectionManager , set alwaysClose=true or call ((SimpleHttpConnectionManager)client.getHttpConnectionManager()).shutdown() after each request.

For MultiThreadedHttpConnectionManager , ensure proper configuration: increase http.connection-manager.max-per-host for high‑QPS hosts and adjust http.connection-manager.max-total if many distinct hosts are accessed.

Set appropriate timeouts: client.getHttpConnectionManager().getParams().setConnectionTimeout() , setSoTimeout() , and setConnectionManagerTimeout() to avoid indefinite waits.

Recommended Configuration Example

HttpClient client = new HttpClient(new MultiThreadedHttpConnectionManager());
client.getParams().setConnectionManagerTimeout(5000L);
client.getHttpConnectionManager().getParams().setConnectionTimeout(3000);
client.getHttpConnectionManager().getParams().setSoTimeout(10000);
client.getHttpConnectionManager().getParams().setMaxTotalConnections(100);
client.getHttpConnectionManager().getParams().setDefaultMaxConnectionsPerHost(10);

By reusing the client, properly closing connections, and tuning the pool parameters, the CLOSE_WAIT count drops dramatically and the service regains stability.