Why Does ThreadPoolExecutor Queue Tasks When Max Threads Are Still Idle?

How does a thread pool run tasks?

When a task is submitted to a ThreadPoolExecutor, the JDK follows four simple steps:

Core threads not full : a new thread is created immediately, even if other core threads are idle.

Core threads full : the task is offered to the work queue and waits.

Queue full and pool size < maximumPoolSize : a non‑core (maximum) thread is created to handle the task.

Queue full and pool size = maximumPoolSize : the rejection policy is applied.

Why queue before creating max threads?

The pool is designed as a rate‑limiter and buffer, not to immediately saturate the CPU. Creating a new thread is expensive: each thread allocates ~1 MB of stack memory and requires a system call. If thousands of tasks arrive at once, spawning thousands of threads can cause severe CPU thrashing and even OOM.

Having many threads does not guarantee higher throughput because context‑switch overhead grows with thread count, especially when the number of threads far exceeds the number of CPU cores.

Therefore, using a queue as an intermediate buffer is a prudent strategy.

Analogy

Think of a bank with three regular service windows (core threads). When a fourth customer arrives, the manager does not immediately open a temporary window; instead, the customer takes a seat in the waiting area (the queue). Only when the waiting area is full and the lobby becomes crowded does the manager call in extra staff (max threads). If both the waiting area and staff are exhausted, the bank displays a “full” sign (rejection).

Can we create max threads first?

For latency‑sensitive services like Tomcat, the default JDK logic can be problematic because a request may sit in the queue even though CPU capacity is idle. Tomcat therefore customizes the queue to force thread creation before queuing.

Tomcat extends LinkedBlockingQueue with a TaskQueue and overrides offer:

@Override
public boolean offer(Runnable o) {
    // 1. If pool size == maximum, fall back to normal queueing
    if (parent.getPoolSize() == parent.getMaximumPoolSize()) {
        return super.offer(o);
    }
    // 2. If submitted tasks <= current threads, queue directly
    if (parent.getSubmittedCount() <= parent.getPoolSize()) {
        return super.offer(o);
    }
    // 3. Core logic: if pool size < maximum, pretend the queue is full to force a new thread
    if (parent.getPoolSize() < parent.getMaximumPoolSize()) {
        return false; // trick the executor into creating a non‑core thread
    }
    // 4. Default: queue the task
    return super.offer(o);
}

This makes Tomcat follow the order: core threads full → create max threads → max threads full → queue → queue full → reject.

Practical advice

Many developers instantiate a new LinkedBlockingQueue() without specifying a capacity. Its default capacity is Integer.MAX_VALUE (≈2.1 billion), meaning the queue will virtually never fill, and maximumPoolSize becomes ineffective; tasks will all queue, leading to high latency and possible OOM under load.

Configure thread pools based on the workload:

For high‑concurrency, low‑latency services (e.g., HTTP servers), follow Tomcat’s approach and prioritize thread creation.

For background batch jobs that require high throughput, the JDK default with a reasonably sized bounded queue is safest.