Why APISIX TLS Handshakes Failed: Missing SNI and SSLv2 Pitfalls

Background and Migration Motivation

Our team has used Nginx as the primary reverse‑proxy for over a decade, benefiting from its reliability and efficiency. As the number of deployed systems grew and requirements such as traffic splitting and flow control increased, Nginx configurations became increasingly complex and error‑prone. To simplify management and improve scalability, we decided to replace Nginx with APISIX, an open‑source Apache‑licensed gateway built on Nginx and OpenResty.

Encountered TLS Problem

After switching the test environment from Nginx to APISIX, some internal and external users reported Java client handshake failures when accessing HTTPS endpoints. The Java exception was:

javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

The issue was intermittent, affecting only a subset of applications running in Kubernetes, and could not be reproduced with simple curl commands. Debug logs from both the client and APISIX did not reveal useful clues.

TLS Basics and Handshake Flow

Before diving deeper, we reviewed the TLS protocol stack, which sits below HTTP. The handshake involves a client and server exchanging hello messages, negotiating protocol version and cipher suites, then exchanging certificates and keys to derive the session secret. In production, many optional steps (e.g., client‑certificate verification) are omitted.

HTTPS Request Chain in Our Architecture

The request flow can be split into two segments: client → APISIX (nginx) and APISIX (nginx) → upstream services . Previously, Nginx performed TLS offloading, decrypting traffic before forwarding plain HTTP to back‑ends. We kept the same offloading strategy after migrating to APISIX, so the failure occurred on the client‑to‑APISIX segment.

Initial SNI Investigation

We suspected the problem matched a known issue described in a CSDN article, where the client did not send an SNI (Server Name Indication) extension. The article suggested disabling SNI on the client side to reproduce the failure:

System.setProperty("jsse.enableSNIExtension", "false");

Packet captures on the APISIX side confirmed that the ClientHello lacked the SNI field, and APISIX responded with a fatal internal error.

Understanding SNI

SNI is a TLS extension that allows the client to indicate the intended hostname during the ClientHello phase, enabling the server to present the correct certificate for virtual hosts sharing the same IP address. Both client and server must support SNI for it to work.

On the server side, OpenResty (the Nginx variant used by APISIX) can verify SNI support with: /usr/bin/openresty -V | grep SNI Output confirms that SNI is enabled in the bundled OpenSSL library.

Resolving Missing SNI with Fallback SNI

Rather than modifying every Java client, we configured APISIX to provide a fallback SNI value when the client omits the extension. Adding the following to the APISIX configuration solves the handshake:

fallback_sni: "my.default.domain"  # Used when client does not send SNI

After applying this setting, TLS 1.2 handshakes succeeded even without client‑side SNI.

We also noted that Nginx implicitly handled missing SNI by falling back to the default server block, which worked because a wildcard certificate covered all hostnames.

Remaining SSLv2Hello Issue

Some clients still failed because they attempted the handshake using the legacy SSLv2Hello format. The client code listed supported protocols as:

supportedProtocols = {"SSLv2Hello", "SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2"};

APISIX’s ssl_protocols directive also included SSLv2, SSLv3, and all TLS versions:

ssl_protocols: SSLv2 SSLv3 TLSv1 TLSv1.1 TLSv1.2 TLSv1.3

Testing showed no improvement. The final workaround was to remove the outdated protocols from the client configuration, limiting it to TLS 1.1 and TLS 1.2: supportedProtocols = {"TLSv1.1", "TLSv2"}; This eliminated the SSLv2Hello handshake attempts and restored connectivity.

Conclusion

The gateway serves as the entry point for the entire system, handling cross‑cutting concerns such as TLS termination, CORS, rate limiting, and traffic routing, allowing back‑ends to focus on business logic. This TLS case illustrates how gateway‑level issues span both front‑end and back‑end domains, requiring deep knowledge of security protocols, network tracing, and configuration tuning to resolve.