Investigation and Resolution of HiveServer2 JDBC Connection Failures and GC‑Induced Hang

Developers using JDBC to connect to HiveServer2 (or Spark HiveThriftServer2) for simple queries, complex analytics, and ultra‑complex analyses often face very high session concurrency (500‑1000+), demanding both OLTP‑style throughput and OLAP‑style analysis, which the Hadoop‑based HiveServer2 is not designed to handle.

Two failure symptoms are observed: (1) JDBC connections cannot be established, and (2) connections succeed but no SQL tasks can be executed, forcing a HiveServer2 restart each time.

Analysis reveals two root causes: (1) extremely high concurrency and memory‑intensive SQL cause severe GC pressure on HiveServer2, leading to connection failures; (2) a single large SQL consumes all resource‑pool capacity, blocking all other tasks from acquiring resources.

Reproduction steps include using jstat to sample GC metrics every 10 seconds, which shows severe GC during hangs, and executing a three‑table join (10^9, 5×10^8, 10^8 rows) that stalls in the reduce phase, occupying resources and preventing other jobs from running.

To resolve the issues, HiveServer2 was migrated to a server with abundant CPU and memory, and JVM parameters were expanded and tuned, for example:

-Xms49152m
-Xmx49152m
-Djava.rmi.server.hostname=xxx.xxx.xxx.xxx
-Dcom.sun.management.jmxremote.port=8999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-XX:+UseParNewGC
-XX:ParallelGCThreads=30
-XX:MaxTenuringThreshold=10
-XX:TargetSurvivorRatio=70
-XX:+UseConcMarkSweepGC
-XX:+CMSConcurrentMTEnabled
-XX:ParallelCMSThreads=30
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+CMSClassUnloadingEnabled
-XX:+DisableExplicitGC
-XX:CMSInitiatingOccupancyFraction=70
-XX:+UseCMSCompactAtFullCollection
-XX:CMSFullGCsBeforeCompaction=1
-verbose:gc
-XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:GCLogFileSize=512M
-Xloggc:/data/log/tbds/spark/gc-sparkthrift.log-${timenow}

Parameters such as -Xms / -Xmx should match available memory, and thread‑related settings should not exceed CPU core counts. After applying these optimizations, HiveServer2 stability improved significantly, though its inherent JDBC concurrency limits remain, and extreme session counts may still require restarts and concurrency control.