Understanding HBase Write Path, Memstore Limits, and Preventing RegionServer OOM

The article begins with a concise overview of the HBase write pipeline: a client API call is protobuf‑encoded, sent via the IPC module to the server’s RPC queue, processed by an RPC handler, written first to the WAL, then to the in‑memory memstore, and finally flushed to HFiles on the filesystem.

1. Issues When Writes Are Too Fast

When the memstore size exceeds the configured blockingMemStoreSize , HBase triggers a flush and throws RegionTooBusyException , blocking further writes. The threshold is calculated as hbase.hregion.memstore.flush.size multiplied by hbase.hregion.memstore.block.multiplier (e.g., 256 MiB × 3 = 805 MiB).

private void checkResources() throws RegionTooBusyException {
    // skip meta region
    if (this.getRegionInfo().isMetaRegion()) return;
    if (this.memstoreSize.get() > this.blockingMemStoreSize) {
        blockedRequestsCount.increment();
        requestFlush();
        throw new RegionTooBusyException("Above memstore limit, " +
            "regionName=" + (this.getRegionInfo() == null ? "unknown" : this.getRegionInfo().getRegionNameAsString()) +
            ", server=" + (this.getRegionServerServices() == null ? "unknown" : this.getRegionServerServices().getServerName()) +
            ", memstoreSize=" + memstoreSize.get() +
            ", blockingMemStoreSize=" + blockingMemStoreSize);
    }
}

Typical log entries show the exception and may eventually lead to JVM OutOfMemoryError if the write pressure continues.

RegionTooBusyException: Above memstore limit, regionName=xxxxx ...
java.lang.OutOfMemoryError: Java heap space

2. Consequences of Memstore Saturation

If the global memstore usage exceeds the heap‑based upper limit (default 40% of heap), RegionServer write operations are blocked, the request queue grows, and the process can OOM. HBase does not terminate the process on OOM, leaving the server in a semi‑dead state.

hbase.regionserver.global.memstore.upperLimit=0.4
hbase.regionserver.global.memstore.size=0.4

3. Mitigation Strategies

Several configuration tweaks can reduce the risk of OOM:

Accelerate flushes: adjust hbase.hstore.blockingWaitTime , hbase.hstore.flusher.count , and hbase.hstore.blockingStoreFiles .

Increase compaction threads: hbase.regionserver.thread.compaction.small and hbase.regionserver.thread.compaction.large .

Limit RPC call queue size to provide back‑pressure: hbase.ipc.server.max.callqueue.size=1024*1024*1024 (1 GB).

hbase.hstore.blockingWaitTime = 90000 ms
hbase.hstore.flusher.count = 2
hbase.hstore.blockingStoreFiles = 10

These settings must be applied carefully; some require a server restart, while others (e.g., call‑queue size) can be changed dynamically in cloud‑hosted HBase.

Overall, understanding the write path, monitoring memstore thresholds, and tuning the above parameters are essential to keep HBase RegionServers stable under heavy write workloads.