How to Diagnose and Fix Common HBase RegionServer Crashes

When operating an HBase cluster, engineers often encounter RegionServer crashes, increased write latency, or complete write failures. This article shares real‑world production cases, explains the root‑cause analysis process, and summarizes how to use logs and monitoring tools to build a systematic troubleshooting workflow.

Case 1: Long GC Causing RegionServer Crash

Symptom: Alarm indicating RegionServer process exit.

Step 1 – Locate the Cause: The issue is not visible in metrics; you must search the RegionServer logs for keywords such as "a long garbage collecting pause" or "ABORTING region server". Example log excerpts:

<code>2019-06-14T17:22:02.054 WARN [JvmPauseMonitor] Detected pause in JVM or host machine (eg GC): pause of approximately 20542ms
GC pool 'ParNew' had collection(s): count=1 time=0ms
GC pool 'ConcurrentMarkSweep' had collection(s): count=2 time=20898ms
WARN [regionserver60020.periodicFlusher] We slept 20936ms instead of 100ms, likely due to a long garbage collecting pause</code>

Step 2 – GC Analysis: The CMS collector can trigger a Full GC in two scenarios: Concurrent Mode Failure and Promotion Failure.

Step 3 – Fault Analysis: A concurrent mode failure forces the JVM into a stop‑the‑world pause, causing the RegionServer's session with Zookeeper to time out. Zookeeper then instructs the Master to evict the dead RegionServer.

Step 4 – Solution: Adjust the CMS collector to run earlier, e.g., set

CMSInitiatingOccupancyFraction=60

. Also verify that BlockCache off‑heap mode is enabled and that JVM startup parameters are reasonable.

Case 2: Large Scan Result Leading to RegionServer Crash

Symptom: RegionServer process exits.

Step 1 – Locate the Cause: Search logs for "abort" or "OutOfMemoryError". Example log line:

<code>java.lang.OutOfMemoryError: Requested array exceeds VM limit</code>

Step 2 – Source Confirmation: The error occurs when a scan returns an excessively large result, causing the JVM to request an array larger than the allowed maximum (Integer.MAX_VALUE‑2).

Step 3 – Fault Analysis: This is effectively an HBase bug; the system should not allocate arrays beyond the JVM limit. It may also stem from improper client usage.

Step 4 – Solution: Limit scan results on the server side with

hbase.server.scanner.max.result.size

or on the client side with

scan.setMaxResultSize(...)

.

Case 3: HDFS Decommission Causing Write Exceptions

Symptom: Some write requests time out while HDFS DataNodes are being retired.

Step 1 – Locate the Cause: During decommission, node I/O load spikes. Check RegionServer logs for exceptions such as:

<code>WARN [ResponseProcessor] ... Bad response ERROR for block ...
INFO [sync.0] wal.FSHLog: Slow sync cost: 13924 ms</code>

HLog sync takes excessive time, indicating write blocking.

Step 2 – Analysis: Simultaneous DataNode retirements increase bandwidth and I/O pressure, slowing down block writes and causing HLog flush timeouts, which leads to write accumulation and timeouts under heavy load.

Step 3 – Solution: Perform DataNode retirements during low‑traffic periods and retire nodes sequentially rather than in parallel.

General HBase Operations Troubleshooting Workflow

Production incidents serve as a valuable learning resource for operations engineers. The troubleshooting process consists of three stages:

Problem Localization: Identify the trigger using monitoring metrics (CPU, I/O, bandwidth, RegionServer TPS, latency, queue lengths, MemStore usage, BlockCache hit rate) and log analysis (search for "Exception", "ERROR", "WARN").

Problem Analysis: Combine metric observations with system architecture knowledge to infer the root cause.

Problem Resolution: Apply targeted fixes based on the analysis, and optionally verify the fix by reviewing source code.

When logs alone are insufficient, searching the web (StackOverflow, HBase forums) or contacting the community can provide additional insight.

Below is a visual overview of the localization process:

Reference: "HBase: Principles and Practice".