Skip to content

HTTP Requests#

HTTP is the lingua franca of web APIs. Python offers layers from stdlib urllib to ergonomic third-party clients (requests, httpx). Production code demands explicit timeouts, status checking, retries with backoff, and secure secret handling — not bare urlopen calls.

How to use this page

Use httpx or requests for application code; know urllib for stdlib-only environments. Pair with Error Handling, Managing Secrets, and Threading for parallel downloads.

At a glance
Track Python Advanced → Network and Web Programming
Sections 11 major topics
Outline Use the right-hand TOC to jump

Topics: HTTP quick reference · Stdlib: urllib.request · requests — ergonomic sync client · httpx — modern client (sync and async) · Authentication patterns · Retries with exponential backoff · Handling responses safely · Parallel requests · … (+3 more)

  1. HTTP quick reference
  2. Stdlib: urllib.request
  3. requests — ergonomic sync client
  4. httpx — modern client (sync and async)
  5. Authentication patterns
  6. Retries with exponential backoff
  7. Handling responses safely
  8. Parallel requests
  9. SSL/TLS verification
  10. Error mapping
  11. Testing HTTP code

HTTP quick reference#

Method Typical use Idempotent
GET Fetch resource Yes
POST Create / submit No
PUT Replace resource Yes
PATCH Partial update No
DELETE Remove resource Yes
HEAD Metadata only Yes
Status range Meaning
2xx Success
3xx Redirect
4xx Client error (bad request, auth, not found)
5xx Server error

Stdlib: urllib.request#

import json
import urllib.request
import urllib.error

url = "https://api.example.com/users/1"
req = urllib.request.Request(
    url,
    headers={"Accept": "application/json", "User-Agent": "myapp/1.0"},
    method="GET",
)

try:
    with urllib.request.urlopen(req, timeout=10.0) as resp:
        status = resp.status
        body = resp.read().decode("utf-8")
        data = json.loads(body)
except urllib.error.HTTPError as exc:
    print(exc.code, exc.reason)  # 4xx/5xx
except urllib.error.URLError as exc:
    print(exc.reason)  # DNS, connection refused, timeout

Always pass timeout

Without it, a hung server blocks forever. Default is no timeout in urllib.

POST JSON with urllib#

import json
import urllib.request

payload = json.dumps({"name": "Ada"}).encode("utf-8")
req = urllib.request.Request(
    "https://api.example.com/users",
    data=payload,
    headers={"Content-Type": "application/json"},
    method="POST",
)
with urllib.request.urlopen(req, timeout=10.0) as resp:
    created = json.loads(resp.read())

requests — ergonomic sync client#

import requests

response = requests.get(
    "https://api.example.com/users/1",
    timeout=(3.0, 10.0),  # (connect, read)
    headers={"Accept": "application/json"},
)
response.raise_for_status()  # raises HTTPError for 4xx/5xx
data = response.json()

Sessions and connection pooling#

import requests

with requests.Session() as session:
    session.headers.update({"Authorization": f"Bearer {token}"})
    session.get("https://api.example.com/me", timeout=10.0)
    session.get("https://api.example.com/items", timeout=10.0)

Session reuses TCP connections — significant performance win for many requests to the same host.

POST form and files#

import requests

# Form-urlencoded
requests.post(url, data={"q": "python"}, timeout=10.0)

# Multipart file upload
with open("report.pdf", "rb") as f:
    requests.post(url, files={"file": f}, timeout=30.0)

httpx — modern client (sync and async)#

import httpx

with httpx.Client(timeout=10.0) as client:
    r = client.get("https://api.example.com/health")
    r.raise_for_status()

Async requests#

import asyncio
import httpx

async def fetch_all(urls: list[str]) -> list[httpx.Response]:
    async with httpx.AsyncClient(timeout=10.0) as client:
        tasks = [client.get(url) for url in urls]
        return await asyncio.gather(*tasks)

asyncio.run(fetch_all(["https://a.com", "https://b.com"]))
Library Sync Async HTTP/2
urllib Yes No No
requests Yes No* No
httpx Yes Yes Optional

* Third-party async wrappers exist; native async is httpx.

Authentication patterns#

import os
import httpx

token = os.environ["API_TOKEN"]  # never hardcode — see Managing Secrets

headers = {"Authorization": f"Bearer {token}"}

# Basic auth
auth = httpx.BasicAuth(username="user", password=os.environ["API_PASSWORD"])
client = httpx.Client(auth=auth, timeout=10.0)

Load tokens from environment or secret stores — Managing Secrets.

Retries with exponential backoff#

import time
import httpx

def get_with_retry(url: str, max_attempts: int = 3) -> httpx.Response:
    delay = 1.0
    for attempt in range(1, max_attempts + 1):
        try:
            r = httpx.get(url, timeout=10.0)
            if r.status_code in {429, 500, 502, 503, 504}:
                raise httpx.HTTPStatusError("retryable", request=r.request, response=r)
            r.raise_for_status()
            return r
        except (httpx.TimeoutException, httpx.HTTPStatusError):
            if attempt == max_attempts:
                raise
            time.sleep(delay)
            delay *= 2

For production, prefer urllib3.util.retry.Retry (used by requests) or dedicated libraries (tenacity).

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
import requests

retry = Retry(total=3, backoff_factor=0.5, status_forcelist=[502, 503, 504])
adapter = HTTPAdapter(max_retries=retry)
session = requests.Session()
session.mount("https://", adapter)

Handling responses safely#

import httpx

r = httpx.get(url, timeout=10.0)

# Content negotiation
if "application/json" in r.headers.get("content-type", ""):
    payload = r.json()
else:
    text = r.text

# Streaming large downloads
with httpx.stream("GET", url, timeout=30.0) as r:
    r.raise_for_status()
    with open("large.bin", "wb") as f:
        for chunk in r.iter_bytes(chunk_size=65536):
            f.write(chunk)

Write streamed downloads to disk with patterns from File IO.

Parallel requests#

Thread pool (I/O-bound)#

from concurrent.futures import ThreadPoolExecutor
import httpx

urls = ["https://api.example.com/item/1", "https://api.example.com/item/2"]

def fetch(url: str) -> dict:
    r = httpx.get(url, timeout=10.0)
    r.raise_for_status()
    return r.json()

with ThreadPoolExecutor(max_workers=8) as pool:
    results = list(pool.map(fetch, urls))

See Threading for synchronization when aggregating results.

Rate limiting#

Respect Retry-After headers and API quotas. Use semaphores or token buckets to cap concurrent requests.

SSL/TLS verification#

import httpx

# Default: verify=True (certificate validation enabled)
r = httpx.get("https://example.com", verify=True)

# Custom CA bundle
r = httpx.get(url, verify="/path/to/ca-bundle.crt")

Never disable verification in production

verify=False invites man-in-the-middle attacks. Fix certificates or trust stores instead.

Error mapping#

Exception Source Typical cause
httpx.TimeoutException httpx Connect/read timeout exceeded
httpx.HTTPStatusError httpx 4xx/5xx after raise_for_status()
requests.HTTPError requests Same
urllib.error.HTTPError urllib HTTP error response
urllib.error.URLError urllib Network-level failure
json.JSONDecodeError json Body is not valid JSON

Wrap API calls with domain-specific exceptions — patterns in Error Handling.

Testing HTTP code#

import httpx
import pytest

def test_get_user(httpx_mock):
    httpx_mock.add_response(json={"id": 1, "name": "Ada"})
    data = fetch_user(1)
    assert data["name"] == "Ada"

Use pytest-httpx, responses, or unittest.mock.patch — covered in Unit Testing.

Interview traps (quick reference)#

Trap What goes wrong Safe approach
No timeout Thread hangs indefinitely Always set connect + read timeouts
Ignoring HTTP status Treat 404 body as success raise_for_status() or explicit checks
Hardcoded API keys Leaked in git Environment variables / secret manager
verify=False shortcut MITM vulnerability Fix cert chain
New connection per request Slow, socket exhaustion Session / Client reuse
response.json() on empty body JSONDecodeError Check status and content-type
Unbounded parallel GETs Rate limit / IP ban Semaphore + backoff

Mental model checklist#

  1. What is the difference between connect and read timeout?
  2. Why use requests.Session() for multiple calls?
  3. When should you stream a response instead of .content?
  4. Which HTTP status codes are commonly retryable?
  5. How do you pass auth headers without committing secrets?
  6. When is httpx.AsyncClient preferable to threads?

What's next#

Topic Page
Parallel fetches Threading
Saving responses File IO
API keys Managing Secrets
Mocking HTTP Unit Testing