Dynamic Thread Pool: Monitoring, Alerting, and Runtime Parameter Adjustment

In backend development, thread pools are widely used to improve throughput and response time, but configuring their parameters is difficult and often leads to overload, service unavailability, or memory leaks. A dynamic thread pool can change core parameters at runtime without redeploying the application and provide monitoring and alerting to detect abnormal situations early.

What is a dynamic thread pool? It is a thread pool whose core parameters (core size, max size, queue capacity) can be modified in real time and whose state can be observed and warned, allowing developers to quickly sense and adjust to workload changes.

Why is it needed? Three main pain points are identified:

Runtime status is hard to perceive – developers cannot know thread counts, queue backlog, or pool exhaustion until an incident occurs.

Thread rejections are difficult to locate – short‑lived rejections leave no stack trace, making root‑cause analysis hard.

Parameter adjustments are slow – the typical "modify → package → approve → release" cycle prolongs risk exposure.

The article surveys existing dynamic thread‑pool solutions and finds they are tightly coupled with middleware, lack flexibility, and do not support custom actions such as automatic stack printing or one‑click queue clearing.

Overall solution combines pool monitoring, exception alerting, automatic stack capture, and dynamic parameter refresh into a single component tailored to the company’s middleware and business characteristics.

Thread‑pool monitoring and alerting

Key steps:

Register each thread pool in a ThreadPoolManager at application startup.

Query ThreadPoolExecutor for snapshot data (core size, max size, queue size, active count, task counts).

Export metrics to Micrometer/Prometheus/Grafana or use the internal pfinder metric system.

Wrap the rejection policy to emit an alarm before the actual rejection.

public class ThreadPoolManager {
    private static final ConcurrentHashMap
REGISTER_MAP_BY_NAME = new ConcurrentHashMap<>();
    private static final ConcurrentHashMap
REGISTER_MAP_BY_EXECUTOR = new ConcurrentHashMap<>();
    public static void registerExecutor(String threadPoolName, Executor executor) {
        REGISTER_MAP_BY_NAME.putIfAbsent(threadPoolName, executor);
        REGISTER_MAP_BY_EXECUTOR.putIfAbsent(executor, threadPoolName);
    }
    public static Executor getExecutorByName(String threadPoolName) {
        return REGISTER_MAP_BY_NAME.get(threadPoolName);
    }
    public static String getNameByExecutor(Executor executor) {
        return REGISTER_MAP_BY_EXECUTOR.get(executor);
    }
    public static Set
getAllExecutorNames() {
        return REGISTER_MAP_BY_NAME.keySet();
    }
}

public static void monitorRegister() {
    log.info("===\> monitor register start...");
    Set
allExecutorNames = ThreadPoolManager.getAllExecutorNames();
    allExecutorNames.forEach(executorName -> {
        if (!monitorThreadPool.contains(executorName)) {
            monitorThreadPool.add(executorName);
            Executor executor = ThreadPoolManager.getExecutorByName(executorName);
            collect(executor, executorName);
        }
    });
    log.info("===\> monitor register end...");
}

Metrics such as corePoolSize , maximumPoolSize , and queue.size() are reported via pfinder gauges.

Automatic stack‑trace printing on rejection

When a rejection occurs, the custom RejectInvocationHandler triggers an alarm and prints the full JVM stack trace of all threads, helping pinpoint the cause.

@Slf4j
public class RejectInvocationHandler implements InvocationHandler {
    private final Object target;
    @Value("${jtool.pool.reject.alarm.key}")
    private String key;
    public RejectInvocationHandler(Object target) { this.target = target; }
    @Override
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        ExecutorService executor = (ExecutorService) args[1];
        if ("rejectedExecution".equals(method.getName())) {
            try { rejectBefore(executor); } catch (Exception e) { log.error("Exception while do rejectBefore", e); }
        }
        return method.invoke(target, args);
    }
    private void rejectBefore(ExecutorService executor) { rejectAlarm(executor); }
    private void rejectAlarm(ExecutorService executor) {
        String alarmKey = Objects.nonNull(key) ? key : ThreadPoolConst.UMP_ALARM_KEY;
        ThreadPoolExecutor tp = (ThreadPoolExecutor) executor;
        String threadPoolName = ThreadPoolManager.getNameByExecutor(tp);
        String errorMsg = String.format("===\> 线程池拒绝报警 key: [%s], cur executor: [%s], core size: [%s], max size: [%s], queue size: [%s]",
                alarmKey, threadPoolName, tp.getCorePoolSize(), tp.getMaximumPoolSize(), tp.getQueue().size());
        log.error(errorMsg);
        Profiler.businessAlarm(alarmKey, errorMsg);
    }
}

public static void printThreadStack(Executor executor) {
    if (!CommonProperty.logPrintFlag) { log.info("===\> 线程池堆栈打印关闭"); return; }
    logger.info("\n=================> 线程池拒绝堆栈打印start");
    Map
allStackTraces = Thread.getAllStackTraces();
    allStackTraces.forEach((thread, stack) -> {
        StringBuilder sb = new StringBuilder();
        sb.append(String.format("\n线程：[%s] 时间: [%s]\n", thread.getName(), new Date()));
        sb.append(String.format("\tjava.lang.Thread.State: %s\n", thread.getState()));
        for (StackTraceElement e : stack) { sb.append("\t\t").append(e.toString()).append("\n"); }
        logger.info(sb.toString());
    });
    logger.info("==============>> end");
}

A helper ThreadLogAnalyzer can parse the generated logs to aggregate thread states and hot‑spot methods.

public class ThreadLogAnalyzer {
    public static void main(String[] args) {
        String logFilePath = "/path/to/reject.monitor.log";
        String threadPoolNameLike = "simpleTestExecutor";
        // parsing logic omitted for brevity
    }
}

Dynamic parameter refresh

Only three core parameters usually need runtime changes: corePoolSize , maximumPoolSize , and queueCapacity . The ThreadPoolExecutor provides setCorePoolSize and setMaximumPoolSize . For queue capacity, a custom mutable queue (e.g., VariableLinkedBlockingQueue) can be used.

@LafValue("jtool.pool.refresh")
public void refresh(@JsonConverter List
threadPoolProperties) {
    String jsonString = JSON.toJSONString(threadPoolProperties);
    log.info("===\> refresh thread pool properties [{}]", jsonString);
    threadPoolProperties = JSONObject.parseArray(jsonString, ThreadPoolProperties.class);
    refresh(threadPoolProperties);
}
public static boolean refresh(List
threadPoolProperties) {
    if (Objects.isNull(threadPoolProperties)) { log.warn("refresh param is empty!"); return false; }
    threadPoolProperties.forEach(prop -> {
        String name = prop.getThreadPoolName();
        Executor executor = ThreadPoolManager.getExecutorByName(name);
        if (executor == null) { log.warn("Register not find this executor: {}", name); return; }
        refreshExecutor(executor, name, prop);
        log.info("Refresh thread pool finish, threadPoolName: [{}]", name);
    });
    return true;
}

In a clustered deployment, a global configuration center (e.g., ducc) can push the new parameters to all nodes, each node retrieves the corresponding pool from ThreadPoolManager and applies the changes.

Conclusion

While thread pools boost system throughput, improper configuration creates hidden risks such as unobservable runtime status, hard‑to‑trace rejections, and slow parameter tuning. By adopting a dynamic thread‑pool approach—monitoring, alerting, automatic stack capture, and live parameter refresh—the article demonstrates how to mitigate these risks and improve backend stability.