Threading#

CPython's threading module lets you run multiple threads of control inside one process. Threads share memory, which makes communication cheap — but the Global Interpreter Lock means only one thread executes Python bytecode at a time. Threading shines for I/O-bound work (network, disk, sleeps); for CPU-bound parallelism, prefer multiprocessing or native extensions that release the GIL.

How to use this page

Start with Thread and Lock, then graduate to concurrent.futures.ThreadPoolExecutor. Pair with Global Interpreter Lock for CPython internals and Error Handling for exception propagation across threads.

At a glance


Track	Python Advanced → Concurrency and Parallelism
Sections	9 major topics
Outline	Use the right-hand TOC to jump

Topics: Threading vs other concurrency models · Creating and managing threads · Synchronization primitives · queue.Queue — thread-safe FIFO · ThreadPoolExecutor — preferred API · Thread-local storage · Exceptions and error handling across threads · Common pitfalls with the GIL · … (+1 more)

Threading vs other concurrency models
Creating and managing threads
Synchronization primitives
queue.Queue — thread-safe FIFO
ThreadPoolExecutor — preferred API
Thread-local storage
Exceptions and error handling across threads
Common pitfalls with the GIL
Thread safety of built-in types

Threading vs other concurrency models#

Model	Module	Best for	Shared memory	GIL impact
Threads	`threading`, `concurrent.futures`	I/O-bound, blocking APIs	Yes	Limits CPU-bound Python
Processes	`multiprocessing`	CPU-bound Python work	No (IPC needed)	Bypasses GIL
Async	`asyncio`	Many concurrent I/O waits	Yes (single thread)	No thread contention
Native code	NumPy, `cryptography`, etc.	Heavy numeric/crypto	Varies	Often releases GIL

import time
from threading import Thread

def fetch(url: str) -> None:
    time.sleep(0.5)  # simulates network I/O — GIL released during sleep
    print(f"done: {url}")

threads = [Thread(target=fetch, args=(f"https://api/{i}",)) for i in range(4)]
for t in threads:
    t.start()
for t in threads:
    t.join()

Creating and managing threads#

Basic `Thread`#

from threading import Thread
import time

def worker(name: str, delay: float) -> str:
    time.sleep(delay)
    return f"{name} finished"

t = Thread(target=worker, args=("A", 1.0), name="worker-a")
t.start()
t.join(timeout=2.0)          # wait up to 2 seconds
print(t.is_alive())          # False after join completes

Parameter	Purpose
`target`	Callable to run in the new thread
`args`	Positional arguments tuple
`kwargs`	Keyword arguments dict
`name`	Debug-friendly thread name
`daemon`	If `True`, thread dies when main exits

Subclassing `Thread`#

from threading import Thread

class DownloadThread(Thread):
    def __init__(self, url: str) -> None:
        super().__init__(name=f"dl-{url}")
        self.url = url
        self.result: bytes | None = None

    def run(self) -> None:
        # Only override run(), not __init__ logic that touches self before start()
        self.result = b"..."  # fetch self.url

t = DownloadThread("https://example.com")
t.start()
t.join()

Prefer target= over subclassing

Subclass when you need state on the thread object; otherwise Thread(target=fn, args=...) is simpler and easier to test.

Daemon threads#

from threading import Thread
import time

def background_logger() -> None:
    while True:
        time.sleep(1)
        print("tick")

t = Thread(target=background_logger, daemon=True)
t.start()
# Main exits → daemon thread is killed abruptly (no finally blocks guaranteed)

Daemon threads are fine for best-effort background tasks. Never rely on them for cleanup or flushing buffers.

Synchronization primitives#

When threads mutate shared state, coordinate access with locks and higher-level primitives.

`Lock` — mutual exclusion#

from threading import Lock

counter = 0
lock = Lock()

def increment() -> None:
    global counter
    with lock:
        counter += 1

Without the lock, counter += 1 is not atomic in Python (read-modify-write race).

`RLock` — reentrant lock#

from threading import RLock

lock = RLock()

def outer() -> None:
    with lock:
        inner()

def inner() -> None:
    with lock:  # same thread can acquire again
        pass

Use RLock when the same thread may re-enter guarded code (e.g., recursive methods).

`Semaphore` — counting access#

from threading import Semaphore
from concurrent.futures import ThreadPoolExecutor

pool_limit = Semaphore(3)  # max 3 concurrent connections

def limited_fetch(url: str) -> str:
    with pool_limit:
        return f"data from {url}"

with ThreadPoolExecutor(max_workers=10) as pool:
    results = list(pool.map(limited_fetch, urls))

`Event` — signal / wait#

from threading import Event, Thread
import time

ready = Event()

def waiter() -> None:
    ready.wait()  # blocks until set()
    print("go!")

def setter() -> None:
    time.sleep(1)
    ready.set()

Thread(target=waiter).start()
Thread(target=setter).start()

`Condition` — complex wait/notify#

from collections import deque
from threading import Condition, Thread

queue: deque[str] = deque()
cond = Condition()

def producer() -> None:
    with cond:
        queue.append("item")
        cond.notify()

def consumer() -> None:
    with cond:
        while not queue:
            cond.wait()
        item = queue.popleft()

For producer-consumer queues, prefer queue.Queue — it wraps a Condition internally and is battle-tested.

`queue.Queue` — thread-safe FIFO#

from queue import Queue, Empty
from threading import Thread

q: Queue[str] = Queue(maxsize=100)

def producer() -> None:
    for i in range(10):
        q.put(f"job-{i}")

def consumer() -> None:
    while True:
        try:
            item = q.get(timeout=1.0)
        except Empty:
            break
        process(item)
        q.task_done()

Thread(target=producer).start()
Thread(target=consumer).start()
q.join()  # blocks until all task_done() called

Method	Behavior
`put(item, block=True, timeout=None)`	Enqueue; blocks if `maxsize` full
`get(block=True, timeout=None)`	Dequeue; blocks if empty
`task_done()`	Mark item processing complete
`join()`	Wait until all items processed

`ThreadPoolExecutor` — preferred API#

from concurrent.futures import ThreadPoolExecutor, as_completed

urls = ["https://a.com", "https://b.com", "https://c.com"]

with ThreadPoolExecutor(max_workers=4) as executor:
    futures = {executor.submit(fetch, url): url for url in urls}

    for future in as_completed(futures):
        url = futures[future]
        try:
            data = future.result()  # re-raises exception from worker
        except OSError as exc:
            print(f"{url} failed: {exc}")

Pattern	When to use
`executor.map(fn, items)`	Same function, many inputs, preserve order
`executor.submit(fn, *args)`	Heterogeneous tasks, need `Future` handles
`as_completed(futures)`	Process results as they finish

Set max_workers deliberately

For I/O-bound work, min(32, os.cpu_count() + 4) is a reasonable default (used by ThreadPoolExecutor). For connection-limited APIs, cap lower to avoid rate limits.

Thread-local storage#

from threading import local

request_context = local()

def handle_request(request_id: str) -> None:
    request_context.id = request_id
    process()  # reads request_context.id without passing through every call

def process() -> None:
    print(request_context.id)

Each thread gets its own namespace on the local() object — useful for per-request state in web servers.

Exceptions and error handling across threads#

Exceptions in worker threads do not propagate to the parent automatically.

from concurrent.futures import ThreadPoolExecutor

def bad() -> None:
    raise ValueError("worker failed")

with ThreadPoolExecutor() as pool:
    future = pool.submit(bad)
    try:
        future.result()  # raises ValueError here
    except ValueError as exc:
        log.exception("thread failed: %s", exc)

For raw Thread, wrap target in a try/except or use a shared error queue. See Error Handling for logging and re-raise patterns.

Common pitfalls with the GIL#

Scenario	Threads help?	Why
Download 100 URLs	Yes	GIL released during socket I/O
Parse JSON in pure Python on 8 cores	No	One bytecode interpreter active
Resize images with Pillow (C extension)	Often yes	C code releases GIL
Increment shared counter 1M times	Barely	Lock contention + GIL overhead

CPU-bound pure Python: use multiprocessing.Pool or ProcessPoolExecutor. Details: Global Interpreter Lock.

Thread safety of built-in types#

Operation	Atomic?	Notes
`list.append()`	Yes (single bytecode)	Safe for append-only
`dict[key] = val`	Yes (single op)	Not safe for read-modify-write
`counter += 1`	No	Use `Lock` or `queue`
`del d[key]`	No	Can race with reads

Interview trap — \"dict operations are thread-safe\"

Individual single dict operations are atomic, but compound logic (if key in d: d[key] += 1) is not. Use locks or queue for coordinated mutations.

Interview traps (quick reference)#

Trap	What goes wrong	Safe approach
Threads for CPU-bound Python	No speedup, often slower	`multiprocessing` or vectorized C libs
Shared mutable state without locks	Data races, corrupted state	`Lock`, `queue.Queue`, immutable data
Expecting exceptions to bubble from `Thread`	Silent failures	`future.result()` or error queue
Daemon thread for cleanup	Abrupt termination	Non-daemon + `join()` or `atexit`
Unbounded thread creation	Memory exhaustion	`ThreadPoolExecutor` with `max_workers`
`join()` without timeout	Deadlock if worker hangs	`join(timeout=...)` + health checks
Assuming `list` is fully thread-safe	Compound ops race	Lock around multi-step mutations

Mental model checklist#

When does CPython release the GIL during threaded I/O?
What is the difference between Lock and RLock?
Why is queue.Queue preferred over a plain list for producer-consumer?
How do you retrieve exceptions from a ThreadPoolExecutor worker?
When should you choose asyncio over threading?
What happens to daemon threads when the main thread exits?

What's next#

Topic	Page
GIL internals	Global Interpreter Lock
Error propagation	Error Handling
HTTP with threads	HTTP Requests
File I/O in workers	File IO