Thread Pools in Java: ThreadPoolExecutor In Depth

Thread pools are one of those topics that every Java developer needs to understand deeply, not just at a surface level. It comes up in interviews constantly, and it comes up even more in day to day work where getting the configuration wrong can bring down your application under load. This article covers everything: why thread pools exist, how every parameter in ThreadPoolExecutor works, the exact decision process when a task arrives, all rejection policies, the difference between execute and submit, the shutdown lifecycle, and the real interview questions that trip people up.

---

Why Thread Pools Exist

Before you can appreciate what a thread pool does for you, you need to feel the pain of the alternative.

Imagine you are building a web server. Every time a request arrives, you create a brand new thread to handle it, do the work, and let the thread die when the work is done. This sounds reasonable until you think about what actually happens at the operating system level.

Creating a thread is not free. The operating system has to allocate stack memory for that thread, set up a program counter, create registers, and do a bunch of bookkeeping to make the thread schedulable. All of that takes time. For a single request it might feel instantaneous, but if your server is handling thousands of requests per second, you are spending a significant portion of your CPU just creating and destroying threads rather than doing actual work. Thread pools solve this by creating a batch of threads upfront, keeping them alive and ready to pick up tasks, and reusing them over and over. The creation cost is paid once, not on every request.

The second problem is even more dangerous: unbounded thread creation. If you create a thread per request and traffic spikes to ten thousand concurrent requests, you have ten thousand threads. Each thread needs its own stack space. On a typical JVM setup, a single thread might need 1 MB or more of stack space. Ten thousand threads means ten thousand megabytes of stack alone, which is ten gigabytes. Unless you have a very unusual machine, your JVM crashes with an OutOfMemoryError long before that. Thread pools give you a hard ceiling on how many threads can exist at once.

The third problem is context switching. Your CPU cores are the only ones doing real work. If you have eight cores and you have created a hundred threads, those hundred threads are not all running at the same time. The CPU switches between them rapidly, giving each one a tiny slice of time. Every switch requires saving the current thread's state and loading the next one's state. More threads means more context switching overhead and less actual processing. The goal of a well configured thread pool is to have enough threads to keep all CPU cores busy without so many threads that context switching eats your performance alive.

Thread pools solve all three problems simultaneously: they eliminate repetitive thread creation cost, cap maximum memory usage, and keep context switching under control.

---

The Executor Framework

Java wraps thread pools inside a clean hierarchy of interfaces and classes. Understanding the hierarchy matters because you will see these names constantly.

At the very top sits the Executor interface. It has exactly one method: execute(Runnable task). That is it. You give it a task and it runs it somewhere.

ExecutorService extends Executor and adds much richer lifecycle management: methods to submit tasks with return values, shut down the pool, wait for termination, and more.

AbstractExecutorService is a partial implementation of ExecutorService that handles common logic so concrete classes do not have to.

ThreadPoolExecutor extends AbstractExecutorService and is the core implementation you will use directly or indirectly. When you call Executors.newFixedThreadPool() or Executors.newCachedThreadPool(), those factory methods create a ThreadPoolExecutor underneath with specific parameter values.

---

The ThreadPoolExecutor Constructor: Seven Parameters

Here is the full constructor:

ThreadPoolExecutor executor = new ThreadPoolExecutor(
    int corePoolSize,
    int maximumPoolSize,
    long keepAliveTime,
    TimeUnit unit,
    BlockingQueue<Runnable> workQueue,
    ThreadFactory threadFactory,
    RejectedExecutionHandler handler
);

Seven parameters. Every single one matters. Walk through them one at a time.

Parameter 1: corePoolSize

This is the minimum number of threads that will always be kept alive in the pool, even when they have nothing to do. Think of it as your standing army. You pay to keep them ready even during quiet periods.

When you submit the very first task to a brand new pool, the pool creates a new thread to handle it, even if corePoolSize is ten and only one task has arrived. Threads are created lazily. But once created, they stick around permanently. If you set corePoolSize to four, the pool will never have fewer than four live threads once it has started creating them, unless you explicitly configure it to allow core thread timeout.

Why not just set this to the maximum and be done with it? Because keeping threads alive costs memory and has a small overhead. You want to be thoughtful about how many threads sit around burning resources during normal, non peak traffic.

Parameter 2: maximumPoolSize

This is the absolute ceiling on how many threads can exist in the pool at any one time. The pool will never create more threads than this number.

The relationship between corePoolSize and maximumPoolSize defines your elasticity. The pool starts with up to corePoolSize threads for everyday work. During a traffic spike, it can grow beyond that, up to maximumPoolSize. The extra threads beyond the core size are temporary: they get cleaned up when they have been idle for longer than keepAliveTime.

Parameter 3: keepAliveTime and Parameter 4: TimeUnit

These two parameters work together. keepAliveTime is the duration, and TimeUnit is its unit (seconds, milliseconds, minutes, etc.).

When the pool has more threads than corePoolSize and some of those extra threads sit idle for longer than keepAliveTime, the pool terminates them. This prevents you from holding onto resources you no longer need after a traffic spike passes.

By default, this timeout applies only to the excess threads beyond corePoolSize. However, if you call allowCoreThreadTimeOut(true), even the core threads can be terminated after being idle for keepAliveTime. This is useful when you know your application goes through long quiet periods where you want to release all resources.

// Allow core threads to time out after 60 seconds of idleness
executor.allowCoreThreadTimeOut(true);

Parameter 5: workQueue (BlockingQueue)

This is the queue where incoming tasks wait when all core threads are busy. Think of it as a waiting room for tasks that have not yet been picked up.

There are two categories you need to know:

Bounded queues have a fixed capacity. ArrayBlockingQueue is the standard choice. When the queue is full and all threads are busy, new tasks either trigger more thread creation (up to maximumPoolSize) or get rejected.

Unbounded queues like LinkedBlockingQueue with no capacity argument can grow without limit. This sounds convenient but is dangerous. If tasks arrive faster than they can be processed, the queue keeps growing until you run out of heap memory. When you use an unbounded queue, maximumPoolSize effectively becomes meaningless because the queue never fills up, so extra threads beyond corePoolSize are never created.

// Bounded queue - safer, predictable memory usage
BlockingQueue<Runnable> queue = new ArrayBlockingQueue<>(100);

// Unbounded queue - use with caution
BlockingQueue<Runnable> queue = new LinkedBlockingQueue<>();

Parameter 6: ThreadFactory

The thread pool itself creates threads internally. By default it uses a factory that generates generic names like pool-1-thread-1. The ThreadFactory parameter lets you inject your own factory to customize how threads are created.

Why would you want this? Three main reasons:

• Custom names: When you read a thread dump during a production incident, seeing order processor thread 3 is far more useful than pool-2-thread-3.
• Priority: You can set the thread priority for every thread the pool creates.
• Daemon flag: You can make all threads daemon threads, which means they will not prevent the JVM from shutting down.

ThreadFactory factory = new ThreadFactory() {
    private int count = 0;

    @Override
    public Thread newThread(Runnable r) {
        Thread t = new Thread(r);
        t.setName("order processor thread " + (++count));
        t.setDaemon(false);
        t.setPriority(Thread.NORM_PRIORITY);
        return t;
    }
};

If you do not provide one, ThreadPoolExecutor uses Executors.defaultThreadFactory() internally.

Parameter 7: RejectedExecutionHandler

When a task cannot be accepted because all threads are busy, the queue is full, and the pool is at maximum size, the pool has to do something. What it does is determined by the RejectedExecutionHandler. There are four built in options, and you can write your own.

---

The Four Step Task Arrival Decision

This is the most important thing to understand about ThreadPoolExecutor, and it is also the source of the most common interview mistake. When a new task arrives, the pool follows exactly this sequence:

Step 1: If the number of currently running threads is less than corePoolSize, create a new thread immediately and assign this task to it. This happens even if other threads are idle. The pool prefers to grow to core size rather than reuse idle threads. Once core size is reached, this step stops triggering.

Step 2: If the thread count has reached corePoolSize, try to add the task to the queue. If there is space in the queue, the task waits there. No new threads are created.

Step 3: If the queue is full, check if the current thread count is below maximumPoolSize. If yes, create a new thread and assign this task directly to it, bypassing the queue.

Step 4: If the thread count is already at maximumPoolSize and the queue is full, there is nowhere to put this task. The RejectedExecutionHandler is called.

Written as a decision tree:

New task arrives
      |
      v
`Running threads < corePoolSize?`
      |                |
     YES              NO
      |                |
Create new thread    Try to add to queue
      |                |
Task runs         Queue has space?
                       |          |
                      YES        NO
                       |          |
                  Task waits   Running threads &lt; maximumPoolSize?
                                   |                |
                                  YES              NO
                                   |                |
                             Create new thread   REJECT
                             (task bypasses queue)

The Spawn or Queue First Question

Here is the interview question that trips up experienced developers: in a ThreadPoolExecutor, does it spawn extra threads first or fill the queue first?

The answer is: it fills the queue first.

When all core threads are busy, the pool does not immediately create extra threads up to maximumPoolSize. It first tries to put the task into the queue. Only when the queue is full does it create additional threads beyond the core size, and only up to maximumPoolSize.

This surprises people because they assume the pool will use all available thread capacity before falling back to queuing. It does the opposite. If you have corePoolSize = 3, maximumPoolSize = 10, and a queue capacity of 100, and you submit 50 tasks all at once: the pool creates 3 threads, fills the queue with 47 tasks, and never creates threads 4 through 10 unless the queue fills up completely.

The practical implication: if you want the pool to scale up beyond core size quickly, use a small or bounded queue. If you want to absorb bursts without creating extra threads, use a large queue.

---

The Four Rejection Policies

When a task gets rejected (step 4 above), one of four things happens based on your configured handler.

AbortPolicy (Default)

Throws a RejectedExecutionException. The calling code must catch this or the thread submitting the task will crash.

// Default behavior - throws exception on rejection
RejectedExecutionHandler handler = new ThreadPoolExecutor.AbortPolicy();

This is the safest choice from a visibility perspective. Failures are loud and obvious. You will know immediately when your pool is overwhelmed.

DiscardPolicy

Silently drops the rejected task. No exception, no log message, nothing. The task just disappears.

RejectedExecutionHandler handler = new ThreadPoolExecutor.DiscardPolicy();

Use this only when losing tasks is genuinely acceptable, such as processing metrics or non critical events where missing a few data points does not matter.

CallerRunsPolicy

Executes the rejected task in the thread that submitted it, not in any pool thread. If your main thread submitted the task, then your main thread runs the task.

RejectedExecutionHandler handler = new ThreadPoolExecutor.CallerRunsPolicy();

This is an elegant backpressure mechanism. When the pool is overwhelmed, the submitting thread gets busy executing the task itself, which naturally slows down the rate at which new tasks arrive. The pool gets a chance to catch up. The downside is that it can slow down or block your submitting thread, which might be a problem if that thread is handling incoming network connections.

DiscardOldestPolicy

Discards the oldest task sitting at the head of the queue (the one that has been waiting the longest), then retries submitting the new task.

RejectedExecutionHandler handler = new ThreadPoolExecutor.DiscardOldestPolicy();

This makes sense in situations where newer work is more valuable than older work, such as real time data processing pipelines where stale data is useless. However, it is worth noting that this silently drops a task that has already been waiting, which can cause confusing behavior if you are not expecting it.

Custom Handler

You can implement RejectedExecutionHandler yourself to do anything you want: log the rejection, write the task to a database, send an alert, or put the task back into a different queue.

RejectedExecutionHandler customHandler = (task, executor) -> {
    System.err.println("Task rejected: " + task.toString());
    // Could log to monitoring system, write to dead letter queue, etc.
};

---

execute vs submit: What Is the Actual Difference

Both methods submit a task to the pool, but there are important differences.

execute(Runnable task)

execute is defined on the base Executor interface. It accepts a Runnable and returns nothing. If the task throws an unchecked exception, it propagates to the thread's uncaught exception handler.

executor.execute(() -> {
    System.out.println("Running in thread: " + Thread.currentThread().getName());
});

You use execute when you just want to fire and forget. You do not care about a return value.

submit(Runnable task) and submit(Callable<T> task)

submit is defined on ExecutorService. It can accept either a Runnable or a Callable<T>. It returns a Future<T> that you can use to:

• Check if the task is done yet
• Wait for the result (blocking)
• Cancel the task
• Retrieve exceptions that occurred during execution

// Submit a Callable that returns a result
Future<String> future = executor.submit(() -> {
    Thread.sleep(1000);
    return "result from background thread";
});

// Do other work here while task runs...

// Block until result is ready
String result = future.get();
System.out.println(result);

A critical difference: with execute, if your task throws a runtime exception, it goes to the thread's uncaught exception handler. With submit, exceptions are captured inside the Future. They do not surface until you call future.get(), at which point they are wrapped in an ExecutionException. This means if you submit tasks with submit but never call get(), exceptions can silently disappear.

Future<?> future = executor.submit(() -> {
    throw new RuntimeException("something went wrong");
});

// Exception is silent until you call get()
future.get(); // throws ExecutionException wrapping the RuntimeException

Use execute for fire and forget tasks. Use submit when you need the result or need to handle exceptions explicitly.

---

The Shutdown Lifecycle

ThreadPoolExecutor goes through several states during its lifetime. Understanding this is important for graceful shutdown in production services.

Running State

This is the normal state. The pool accepts new tasks and processes queued tasks.

shutdown()

When you call shutdown(), the pool enters the Shutdown state. Two things happen:

1. The pool stops accepting any new tasks. If you try to submit a new task after calling shutdown(), it gets rejected by the RejectedExecutionHandler.
2. The pool continues processing all tasks that are already in progress or waiting in the queue.

Once all queued and in progress tasks complete, the pool moves to the Terminated state and all threads are released.

executor.shutdown();
// No new tasks accepted here
// Existing tasks finish normally

shutdownNow()

This is the force stop option. It does three things:

1. Stops accepting new tasks.
2. Attempts to interrupt all currently running threads by calling interrupt() on each one.
3. Returns a List<Runnable> containing all tasks that were waiting in the queue but never started.

Note that interrupting a thread does not guarantee it stops immediately. If your task does not check Thread.interrupted() or call interruptible blocking operations, it keeps running despite the interrupt. shutdownNow() is a request to stop, not a command.

List<Runnable> unprocessedTasks = executor.shutdownNow();
System.out.println("Tasks that never ran: " + unprocessedTasks.size());

awaitTermination()

After calling shutdown() or shutdownNow(), you often want to wait for the pool to fully finish before your application exits. awaitTermination blocks the calling thread until the pool reaches terminated state or the timeout expires.

executor.shutdown();
if (!executor.awaitTermination(30, TimeUnit.SECONDS)) {
    executor.shutdownNow(); // Force stop if tasks are taking too long
}

This pattern is best practice for graceful shutdown: give tasks a chance to finish normally, then force stop if they take too long.

The Stop State

The stop state is reached after shutdownNow(). The pool does not accept new tasks and it interrupts running threads. Tasks waiting in the queue that have not started yet are not executed.

The progression looks like this:

Running → shutdown() → Shutdown (existing tasks complete) → Terminated
Running → shutdownNow() → Stop (running threads interrupted, queue dumped) → Terminated

---

A Complete Working Example

Here is a concrete example that puts all the pieces together:

import java.util.concurrent.*;

public class ThreadPoolDemo {

    public static void main(String[] args) throws InterruptedException {

        // Custom thread factory with meaningful names
        ThreadFactory factory = r -> {
            Thread t = new Thread(r);
            t.setName("worker-" + t.getId());
            return t;
        };

        // Custom rejection handler that logs rejections
        RejectedExecutionHandler onReject = (task, pool) -> {
            System.out.println("Task rejected: " + task);
        };

        ThreadPoolExecutor executor = new ThreadPoolExecutor(
            2,                              // corePoolSize: always keep 2 threads alive
            4,                              // maximumPoolSize: grow up to 4 during spikes
            30,                             // keepAliveTime: extra threads idle for 30 seconds...
            TimeUnit.SECONDS,               // ...measured in seconds
            new ArrayBlockingQueue<>(2),    // workQueue: hold up to 2 waiting tasks
            factory,                        // threadFactory: use our custom factory
            onReject                        // handler: log rejections instead of throwing
        );

        // Submit 7 tasks to a pool that can handle at most 4 + 2 = 6 at once
        for (int i = 1; i <= 7; i++) {
            final int taskNum = i;
            executor.submit(() -> {
                try {
                    System.out.println("Task " + taskNum
                        + " running on " + Thread.currentThread().getName());
                    Thread.sleep(2000); // simulate work
                    System.out.println("Task " + taskNum + " done");
                } catch (InterruptedException e) {
                    Thread.currentThread().interrupt();
                }
            });
        }

        // Graceful shutdown
        executor.shutdown();
        executor.awaitTermination(1, TimeUnit.MINUTES);
    }
}

Walk through what happens when all seven tasks arrive nearly simultaneously:

• Task 1 arrives: fewer than 2 core threads running, so a new thread (worker) is created and task 1 runs immediately.
• Task 2 arrives: still under corePoolSize, so another new thread is created and task 2 runs immediately.
• Task 3 arrives: corePoolSize reached (2 threads busy). Try the queue. Queue has space. Task 3 waits in queue.
• Task 4 arrives: queue still has space (size 2, one slot used). Task 4 waits in queue.
• Task 5 arrives: queue is full (2/2). Thread count (2) is below maximumPoolSize (4). Create a new thread. Task 5 runs immediately on the new thread, bypassing the queue.
• Task 6 arrives: queue is full. Thread count (3) is below maximumPoolSize (4). Create another new thread. Task 6 runs.
• Task 7 arrives: queue is full. Thread count (4) equals maximumPoolSize. Reject task 7. The custom handler logs the rejection.

Six tasks complete successfully. Task 7 is rejected. The output demonstrates the queue first, then spawn extra threads behavior exactly.

---

How to Choose the Right Pool Size

This is the interview question at the heart of the lecture: an interviewer asks a candidate why they chose corePoolSize = 2. What should the answer be?

The correct answer is: it depends on multiple factors, and you derive the number from a formula combined with constraints.

The CPU Core Formula

A well known starting formula for thread count is:

Number of threads = Number of CPU cores × (1 + Wait time / Service time)

Where:

• Number of CPU cores: the number of processors available, readable via Runtime.getRuntime().availableProcessors()
• Wait time: time the thread spends waiting (for IO, DB calls, external API calls, etc.)
• Service time: time the thread actually spends computing

Let us say you have 8 CPU cores. If a task spends equal time waiting and computing, wait time divided by service time is 1.0, so:

8 × (1 + 1.0) = 16 threads

If the task is almost entirely IO (waiting), say it waits 9 units for every 1 unit of CPU work:

8 × (1 + 9) = 80 threads

This makes intuitive sense. When threads spend most of their time blocked on IO, they are not using CPU anyway. You can have many more threads without causing context switching overhead, because most of them sleep while waiting for IO responses.

For CPU intensive tasks where wait time approaches zero:

8 × (1 + 0) = 8 threads

This also makes sense. If tasks never wait, each thread always needs CPU. Having more threads than cores just causes context switching overhead.

The Memory Constraint

The formula above does not consider memory, which can be a binding constraint. You need to calculate whether you actually have enough JVM memory to support the thread count the formula suggests.

Start with total JVM heap allocation. Subtract what you need for the heap (your objects, caches, etc.), code cache (often around 128 MB), and JVM internal overhead. What remains is available for thread stacks.

Each thread needs stack space. The default stack size is platform dependent but is often around 512 KB to 1 MB. You can control it with the -Xss JVM flag.

Available memory for threads = Total JVM memory - Heap - Code cache - JVM overhead
Max threads by memory = Available memory for threads / Stack size per thread

For example: 2 GB JVM, 1 GB heap, 128 MB code cache, 256 MB JVM overhead, leaving 616 MB for thread stacks. At 5 MB per thread (a generous estimate including stack, PC, and other per thread structures), you can support about 123 threads at most.

The final answer is the minimum of the formula result and the memory result. You take the conservative bound.

Task Nature Matters Most

For purely CPU intensive work, the answer is simple: one thread per CPU core, maybe slightly more to keep cores busy during minor scheduling delays.

For IO intensive work like database queries, HTTP calls, or file reads, you can have many more threads than cores because most threads are sleeping while waiting for the IO to complete. The formula guides you, but load testing under realistic conditions is the definitive answer.

Practical Recommendation

Do not guess. Start with the formula, apply the memory constraint, then load test. Monitor context switching overhead, CPU utilization, and response time percentiles. Tune from there. The number is specific to your workload, your machine, and your task mix.

---

Key Interview Questions and Pitfalls

Q: Does ThreadPoolExecutor spawn extra threads or fill the queue first?

Queue first, always. Extra threads beyond corePoolSize are only created after the queue is full. This trips up a lot of candidates who assume the pool aggressively scales threads before falling back to queuing.

Q: What happens when you submit a task to an executor after calling shutdown()?

The task is rejected. The configured RejectedExecutionHandler is called. By default this throws RejectedExecutionException.

Q: If you use an unbounded LinkedBlockingQueue, what is the effective maximumPoolSize?

It is effectively corePoolSize. Because the queue never fills up (it is unbounded), step 3 of the task arrival decision (create thread beyond core size if queue is full) never triggers. The pool never grows beyond core size.

Q: What is the difference between shutdown() and shutdownNow()?

shutdown() is graceful: stops accepting new tasks, lets queued and running tasks complete. shutdownNow() is forceful: stops accepting new tasks, interrupts running threads, and returns the list of unstarted queued tasks.

Q: With execute(), what happens if the task throws an exception?

The exception propagates to the thread's uncaught exception handler and the thread may terminate. A new thread may be created to replace it. With submit(), exceptions are silently captured inside the Future and only surface when you call future.get().

Q: Why would you use CallerRunsPolicy?

As a backpressure mechanism. When the pool is saturated, making the submitting thread run the task directly slows down task submission naturally, giving the pool time to catch up. It is a self regulating feedback loop.

Q: What are core threads and non core threads?

Core threads are threads within the corePoolSize limit. They stay alive indefinitely (unless allowCoreThreadTimeOut is set). Non core threads are the extra threads created between corePoolSize and maximumPoolSize. They are created only when the queue is full and they expire after being idle for keepAliveTime.

Q: How do you calculate the right thread pool size for a production application?

Start with the formula: CPU cores × (1 + wait time / service time). Apply the JVM memory constraint to ensure you have enough stack space. Load test under realistic conditions. Monitor CPU utilization, context switching, and latency. Tune iteratively. Do not pick a number by intuition alone.

Q: Is ThreadPoolExecutor thread safe?

Yes. The internal state is managed with atomic variables and locks. You can safely submit tasks from multiple threads concurrently.

Pitfall: Creating pools with Executors factory methods without understanding the defaults

Executors.newFixedThreadPool(n) uses an unbounded LinkedBlockingQueue. If tasks pile up, you can run out of heap memory without ever seeing a rejection.

Executors.newCachedThreadPool() sets corePoolSize = 0 and maximumPoolSize = Integer.MAX_VALUE. Under sustained load it can create an effectively unlimited number of threads, crashing the JVM with OutOfMemoryError.

For production use, prefer constructing ThreadPoolExecutor directly with explicitly chosen values for every parameter. The factory methods hide important defaults that can cause production incidents.

---

Summary

Thread pools exist to reuse threads (avoiding repeated creation cost), cap maximum thread count (avoiding memory exhaustion), and reduce context switching (improving throughput). ThreadPoolExecutor gives you seven parameters to tune exactly how this works.

The most important behavioral insight is the four step task arrival sequence: grow to core size first, then queue, then grow to maximum size, then reject. Extra threads beyond core size only appear after the queue fills up, which means an undersized queue is your lever for scaling thread count, and an oversized queue means you will never use the extra thread capacity you configured.

Choose rejection policies deliberately. Abort is safe and visible. Discard is dangerous unless losses are acceptable. CallerRuns provides elegant backpressure. DiscardOldest favors freshness over fairness.

Use execute for fire and forget. Use submit when you need results or exception handling. Always shut down pools gracefully. Size your pools based on CPU core count, task wait to service ratio, and JVM memory constraints, then validate with real load testing.

Master these concepts and you will handle any thread pool interview question thrown at you, and more importantly, you will configure thread pools correctly when it matters in production.