Thread pool deadlock

The problem it solves

Every worker in the pool is occupied. Each one is waiting for a task it submitted to the same pool. The tasks they are waiting for cannot start, because the only threads that could run them are the ones waiting.

No lock is involved. Neither task is wrong on its own. The outer task takes a worker, submits some inner work, and blocks until it finishes — which is a completely ordinary thing to write. The inner task takes a worker and does its job — also ordinary. The bug is the shape: a dependency from a resource back onto itself, which is deadlock with the pool as the resource and no mutex anywhere.

This is probably the most common production deadlock in modern service code, and it is invisible in review because the two halves are usually in different files, written by different people, months apart.

The mechanism

A thread pool is a fixed set of workers pulling from a queue. That fixed size is the whole point — it bounds concurrency, memory and pressure on downstream systems.

The failure needs three things: a task that submits to the pool it runs on, that task waiting for the result, and enough such tasks to occupy every worker. Then the pool’s queue contains work that only a free worker can run, and no worker will ever be free, because being free requires completing a task that is waiting on that queue.

The cycle is the same one from the deadlock page — hold and wait, no preemption, circular wait — with “a worker” as the held resource. The wait-for graph is unusual only in that the thing everyone is waiting on is not held by any single thread, which is why a mutex-oriented deadlock detector reports nothing.

The fix is structural, not numerical: give the inner work a different pool. The dependency then runs from one resource to another rather than back onto itself, and no cycle can form for any pool size. This is the bulkhead pattern — isolate resources so exhaustion in one cannot exhaust another.

What the enumeration shows

Two workers, two outer tasks, two inner tasks, one pool.

6 of 2,500 interleavings deadlock. Read that ratio carefully, because it is the reason this reaches production: 99.8% of orderings complete perfectly. Every test passes. Every staging run is fine. Then the load rises, the window widens, and the service hangs.

Step through a deadlocked one and read the resource-wait line: O1 waits on semaphore done1; O2 waits on semaphore done2; I1 waits on semaphore pool; I2 waits on semaphore pool. Four threads, none of them able to move, and no lock anywhere in the program.

Then the two fixes, which are not equally good.

thread-pool-separate gives the inner work its own pool: 0 deadlocks, and the outer tasks may still occupy every outer worker without it mattering.

thread-pool-bigger doubles the pool to four and adds a fourth caller: 38 of 2,500 deadlock. This is the more instructive variant. Doubling the pool genuinely fixes the original program — and moves the failure to a load nobody has tested. Sizing changes how much traffic closes the cycle; it does not remove the cycle.

The numbers worth carrying

• Pool of 2, two nested callers: 6 of 2,500 interleavings deadlock.
• Pool of 4, four nested callers: 38 of 2,500. The shape scales with you.
• The rule: a pool of n deadlocks when n tasks that each wait on pooled work are running simultaneously. Your safety margin is the gap between your pool size and your peak concurrency of nesting tasks — which is a number nobody tracks.
• Sizing for the common cases: CPU-bound ≈ number of cores; I/O-bound ≈ cores × (1 + wait/compute), which for 90% waiting is ten times the core count. Getting this wrong in the other direction — a huge pool “to be safe” — trades deadlock for memory exhaustion and downstream overload.

Where it breaks down

Nesting is often invisible. The outer task calls a library, which calls a framework, which submits to the common pool. Java’s CompletableFuture defaults to ForkJoinPool.commonPool(), and so do parallel streams — so two unrelated pieces of code can share a pool neither of them named. This is how the bug arrives without anybody writing nested submissions.

ForkJoinPool mitigates but does not eliminate it. Its work-stealing design lets a thread blocked in join() execute other queued tasks instead of idling, which dissolves many of these cycles. It relies on the blocking being done through the pool’s own mechanisms; a task blocked on an external future or a lock is opaque to it and the deadlock returns.

Not deadlocking is not the same as being fine. A pool where outer tasks routinely occupy most workers waiting on inner ones has terrible throughput even when it completes, because most of its capacity is parked.

Timeouts convert it, and do not fix it. A bounded wait turns a hang into a failure and a retry, which is better operationally and leaves the cycle intact — and retries under saturation are their own failure mode, see retry storms.

What people get wrong

“Deadlock requires locks.” This program has none. What it has is a bounded resource and a circular dependency, which is all deadlock ever required.

“We increased the pool size and it went away.” It went away at the load you tested. The variant on this page shows it returning one caller later.

“Just use an unbounded pool.” Then you have no bound on threads, memory or downstream pressure, and you have traded a deadlock for an outage under load. The bound is the feature.

“It only happens under load.” Load widens the window. The cycle exists at any load, and 6 of 2,500 orderings is not a small enough number to rely on.

In production

The diagnosis is straightforward once suspected: a thread dump showing every pool thread parked in a get, join or await, with the pool’s queue non-empty. That combination — all workers waiting, work queued — is unambiguous.

The practices that prevent it:

Never block a pooled task on work submitted to the same pool. If you must nest, use a separate pool for the inner stage. One pool per stage of a pipeline is a good default and gives you bulkheading for free.

Prefer composition to blocking. thenCompose instead of get, await on a promise chain instead of a blocking future — the continuation runs when the result arrives rather than occupying a worker while waiting. This removes the hold-and-wait condition entirely and is why async pipelines do not suffer from this.

Do not use the common pool for anything that blocks. In Java, pass an explicit executor to every CompletableFuture stage that might block. The default is the common pool, and the common pool is sized for CPU-bound work and shared with everything else in the process.

Bound the wait. A get with a timeout turns an indefinite hang into an alert.

The follow-up questions

“Can you deadlock without locks?” — Yes. This program. A bounded resource plus a circular dependency is sufficient.

“Your pool has 10 threads and it hangs. What do you look at?” — A thread dump: are all workers parked waiting, with the queue non-empty?

“Would a bigger pool fix it?” — At the load you tested. Then the fourth-caller variant, which is the answer that shows you understand it is structural.

“How do you avoid it?” — Separate pools per stage, or compose rather than block. Not “make the pool bigger”.