Dynamic Thread Pool: Monitoring, Alerting, and Runtime Parameter Adjustment

In backend development, thread pools are widely used to improve throughput and response time, but configuring their parameters is often difficult, leading to overload, service unavailability, and memory issues. A dynamic thread pool can change core parameters at runtime without redeploying the application and provides monitoring and alerting capabilities.

What is a dynamic thread pool? It allows real‑time modification of core parameters (core size, max size, queue capacity) and monitors its state, enabling developers to detect abnormal situations caused by traffic spikes or business changes and adjust parameters promptly.

Why is it needed? Three main pain points are identified:

Runtime status is hard to perceive, making it difficult to know thread count, queue backlog, or pool exhaustion before incidents occur.

Thread rejections are hard to locate because they happen quickly and stack traces are unavailable.

Adjusting parameters requires the full build‑deploy‑approve‑release cycle, which is slow and risky.

The article surveys existing dynamic thread‑pool solutions and finds they are tightly coupled with middleware, lack flexibility, and do not support features such as automatic stack printing or one‑click queue clearing.

Overall solution combines monitoring, alerting, automatic stack capture, and dynamic parameter refresh into a single component tailored to the company’s middleware and business needs.

Thread‑Pool Monitoring and Alerting

To monitor thread pools, a ThreadPoolManager registers each pool by name and provides lookup methods:

public class ThreadPoolManager {
    private static final ConcurrentHashMap<String, Executor> REGISTER_MAP_BY_NAME = new ConcurrentHashMap<>();
    private static final ConcurrentHashMap<Executor, String> REGISTER_MAP_BY_EXECUTOR = new ConcurrentHashMap<>();
    public static void registerExecutor(String threadPoolName, Executor executor) {
        REGISTER_MAP_BY_NAME.putIfAbsent(threadPoolName, executor);
        REGISTER_MAP_BY_EXECUTOR.putIfAbsent(executor, threadPoolName);
    }
    public static Executor getExecutorByName(String threadPoolName) {
        return REGISTER_MAP_BY_NAME.get(threadPoolName);
    }
    public static String getNameByExecutor(Executor executor) {
        return REGISTER_MAP_BY_EXECUTOR.get(executor);
    }
    public static Set<String> getAllExecutorNames() {
        return REGISTER_MAP_BY_NAME.keySet();
    }
}

Metrics such as core pool size, max size, queue size, and active thread count are collected via the ThreadPoolExecutor API and exported to a monitoring system (e.g., Micrometer → Prometheus → Grafana) using the following registration logic:

public static void monitorRegister() {
    Set<String> allExecutorNames = ThreadPoolManager.getAllExecutorNames();
    allExecutorNames.forEach(name -> {
        if (!monitorThreadPool.contains(name)) {
            monitorThreadPool.add(name);
            Executor executor = ThreadPoolManager.getExecutorByName(name);
            collect(executor, name);
        }
    });
}

public static void collect(Executor executorService, String threadPoolName) {
    ThreadPoolExecutor executor = (ThreadPoolExecutor) executorService;
    String prefix = "thread.pool." + threadPoolName;
    gauge1 = PfinderContext.getMetricRegistry().gauges(prefix)
            .gauge(() -> executor.isShutdown() ? 0 : executor.getCorePoolSize())
            .tags(MetricTag.of("type_dimension", "core_size")).build();
    // similar gauges for max size and queue size …
}

Automatic Stack Trace on Rejection

A custom RejectInvocationHandler intercepts the rejectedExecution method, triggers an alarm, and prints the stack trace of all threads:

@Slf4j
public class RejectInvocationHandler implements InvocationHandler {
    private final Object target;
    @Value("${jtool.pool.reject.alarm.key}")
    private String key;
    public RejectInvocationHandler(Object target) { this.target = target; }
    @Override
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        ExecutorService executor = (ExecutorService) args[1];
        if ("rejectedExecution".equals(method.getName())) {
            try { rejectBefore(executor); } catch (Exception e) { log.error("Exception while do rejectBefore", e); }
        }
        return method.invoke(target, args);
    }
    private void rejectBefore(ExecutorService executor) { rejectAlarm(executor); printThreadStack(executor); }
    private void rejectAlarm(ExecutorService executor) { /* build alarm message and log */ }
    public static void printThreadStack(Executor executor) {
        if (!CommonProperty.logPrintFlag) return;
        Map
all = Thread.getAllStackTraces();
        all.forEach((t, stack) -> {
            StringBuilder sb = new StringBuilder();
            sb.append(String.format("\nThread:[%s] Time:[%s]\n", t.getName(), new Date()));
            sb.append("\tState: ").append(t.getState()).append('\n');
            for (StackTraceElement e : stack) sb.append("\t\t").append(e).append('\n');
            logger.info(sb.toString());
        });
    }
}

Dynamic Parameter Refresh

Runtime adjustment of corePoolSize , maximumPoolSize , and queue capacity is achieved via RPC/HTTP calls that invoke setCorePoolSize and setMaximumPoolSize . For queue capacity, a custom mutable queue (e.g., VariableLinkedBlockingQueue) is used.

@LafValue("jtool.pool.refresh")
public void refresh(@JsonConverter List
threadPoolProperties) {
    String json = JSON.toJSONString(threadPoolProperties);
    log.info("===\> refresh thread pool properties [{}]", json);
    threadPoolProperties = JSONObject.parseArray(json, ThreadPoolProperties.class);
    refresh(threadPoolProperties);
}

public static boolean refresh(List
props) {
    if (props == null) { log.warn("refresh param is empty!"); return false; }
    props.forEach(p -> {
        Executor exec = ThreadPoolManager.getExecutorByName(p.getThreadPoolName());
        if (exec == null) { log.warn("Register not find this executor: {}", p.getThreadPoolName()); return; }
        refreshExecutor(exec, p.getThreadPoolName(), p);
        log.info("Refresh thread pool finish, threadPoolName: [{}]", p.getThreadPoolName());
    });
    return true;
}

The refresh logic can be driven by a global configuration center (e.g., Ducc) to propagate changes across a cluster.

Conclusion

By introducing a dynamic thread‑pool component that provides real‑time monitoring, alerting, automatic stack tracing, and on‑the‑fly parameter updates, the risks associated with static thread‑pool configurations—such as invisible runtime status, hard‑to‑locate rejections, and slow parameter tuning—are significantly mitigated.