Thread pools & Executors

The three moving parts

Every thread pool, in every language, is built from the same three pieces:

• The task — a small object holding what to do, and nothing about who does it. In Java that's a Runnable (returns nothing) or a Callable (returns a value). This is literally the Command pattern: work packaged as an object so it can be stored, queued and passed around.
• The queue — where submitted tasks wait until a worker is free. This is the shock absorber: a burst of 500 tasks doesn't create 500 threads, it creates 500 queue entries.
• The workers — N threads created once at startup, each running the same infinite loop: take() a task from the queue (blocking if it's empty), run it, repeat. A worker never dies between tasks, so the expensive creation cost is paid once per worker, not once per task.

// This is the entire idea of a thread pool, in ~12 lines.
class MiniPool {
    private final BlockingQueue<Runnable> queue = new ArrayBlockingQueue<>(4); // bounded!

    MiniPool(int workers) {
        for (int i = 0; i < workers; i++) {      // create N threads ONCE
            new Thread(() -> {
                while (true) {                   // ...then loop forever
                    Runnable task = queue.take(); // blocks until work arrives
                    task.run();                   // reuse this same thread
                }
            }).start();
        }
    }

    void submit(Runnable task) { queue.put(task); } // you queue WHAT, not WHO
}

Read that while (true) loop again — it is the reason threads get reused. The worker doesn't finish when a task finishes; it goes straight back to the queue for the next one. In the prototype, that's a worker lane flashing ✓ and immediately pulling the next chip off the queue without the threads created counter moving.

Executor: the separation that makes it work

The key design move is separating the task from its execution. Your code says "here is a job" and stops there. The Executor — the pool object — owns the decision of which thread runs it and when. That single interface (execute(Runnable)) means you can swap a 4-thread pool for a 40-thread pool, a scheduled pool, or a same-thread executor in tests, without touching a line of the task code. Command pattern for the what, plus a scheduler for the when.

When a task returns a value you get back a Future (or CompletableFuture, or Python's concurrent.futures.Future) — a receipt you can later call get() on to collect the result, blocking until it's ready. Same as being handed a buzzer at a restaurant: you're free to go do something else, and the buzzer tells you when it's done.

How big should the pool be?

• CPU-bound work (compression, hashing, image resizing) — roughly one worker per CPU core. The threads are genuinely busy computing, so extra threads beyond your core count just add context-switching overhead without doing more work per second.
• I/O-bound work (database calls, HTTP requests, file reads) — the pool can be much larger, often tens or hundreds. Those threads spend nearly all their time blocked waiting for a response, using no CPU at all, so you need many of them in flight to keep the machine busy.
• Measure, don't guess — the classic starting formula is threads ≈ cores × (1 + waitTime / computeTime), but the real answer comes from load-testing your actual workload. Try 1 and 4 in the prototype: one worker drains the queue strictly one task at a time; four finish the same eight tasks in fewer rounds.

Bound the queue, or you've just moved the problem

It's tempting to use an unlimited queue — then nothing is ever refused. But an unbounded queue turns a thread-explosion into a memory explosion: if tasks arrive faster than the pool can finish them, the backlog grows without limit until the process dies, and every queued task gets slower and slower while nobody notices. Always give the queue a maximum size. A full queue is not a failure, it's back-pressure — the system telling its callers "slow down, I'm at capacity" while it's still healthy enough to say so.

When the queue is full, the pool applies a rejection policy: abort (throw an exception so the caller knows), caller-runs (the submitting thread executes the task itself, which naturally slows the producer down), or discard (drop the task, or drop the oldest queued one). In the prototype, switch the chip to 📦 queue: bounded 4 and submit 8 tasks: three fit in the running workers, four fit in the queue, and the leftover turns red ✕ REJECTED.

The standard pool flavours

• Fixed — exactly N threads, forever. The safe default for servers: predictable memory, predictable load on downstream systems.
• Cached — grows on demand and reaps threads idle for 60 seconds. Convenient for short bursts, dangerous under sustained load because it has no upper bound on thread count.
• Single-threaded — one worker, so tasks run strictly one at a time in submission order. Handy when you need serialised access to something without locks.
• Scheduled — runs tasks after a delay or on a repeating timer. The pooled replacement for hand-rolled Timer threads.
• Work-stealing / ForkJoin — each worker has its own deque and steals from busy neighbours when idle. Built for many small, recursive, CPU-bound subtasks (this is what Java's parallel streams run on).

Shutting down gracefully

Pool threads usually keep the program alive, so you must close the pool. shutdown() is the polite one: it stops accepting new tasks but lets everything already submitted finish. shutdownNow() is the impatient one: it tries to interrupt running tasks and returns the list of tasks that never started. Either way, follow up with awaitTermination(timeout) to actually wait for the workers to stop — the usual pattern is shutdown(), wait a while, then shutdownNow() if they're still going.

Reach for a thread pool when...

• Many short tasks arrive continuously — requests, messages, jobs. Creation cost dominates when each task is short, and a pool pays that cost once.
• You need a hard ceiling on concurrency — the pool size is the limit. It protects your own memory and, just as importantly, protects the database or downstream API from being hit by a thousand simultaneous callers.
• Load is spiky — the bounded queue absorbs bursts without spawning threads, and the rejection policy gives you a defined behaviour when even the queue fills.
• You want tasks decoupled from threading — plain Runnables are trivially testable and let you swap the execution strategy (pool size, scheduled, same-thread) without touching the work itself.

Skip it when...

• There's one long-lived background job — a dedicated thread is clearer than a pool of one.
• Tasks block on each other — if pooled tasks wait for other tasks in the same pool you're one step from starvation deadlock; use separate pools or restructure with Producer–Consumer style handoff.
• Your runtime already gives you cheap threads — with goroutines or Java virtual threads you can go thread-per-task and cap concurrency with a semaphore instead.
• The work is trivially fast — running a microsecond of work through a queue costs more in handoff than it saves.