File IO

Advanced file I/O goes beyond open() and read() — covering memory-mapped files, atomic writes, temporary files, high-level tree operations, and performance-conscious patterns for large data. Build on File Handling for pathlib, encodings, and JSON/CSV basics.

How to use this page

Master pathlib and context managers from File Handling first. Use Error Handling for OSError hierarchies and Managing Secrets when reading credential files.

At a glance

Track	Python Advanced → File and Data Handling
Sections	13 major topics
Outline	Use the right-hand TOC to jump

Topics: pathlib advanced patterns · Buffering and performance · Memory-mapped files (mmap) · Atomic and safe writes · tempfile module · shutil — high-level file operations · os and low-level I/O · io module — in-memory streams · … (+5 more)

1. pathlib advanced patterns
2. Buffering and performance
3. Memory-mapped files (mmap)
4. Atomic and safe writes
5. tempfile module
6. shutil — high-level file operations
7. os and low-level I/O
8. io module — in-memory streams
9. Context managers and contextlib
10. Working with large text and CSV
11. File descriptors and with pitfalls
12. Permissions and security … and 1 more sections

---

pathlib advanced patterns

from pathlib import Path

config = Path.home() / ".myapp" / "config.toml"
config.parent.mkdir(parents=True, exist_ok=True)

# Resolve symlinks and absolute path
resolved = Path("data/../data/input.csv").resolve()

# Stat and metadata
stat = resolved.stat()
print(stat.st_size, stat.st_mtime)

# Atomic-ish replace (same filesystem)
tmp = resolved.with_suffix(".tmp")
tmp.write_text("new content", encoding="utf-8")
tmp.replace(resolved)  # os.rename semantics on POSIX

Method	Use case
mkdir(parents=True, exist_ok=True)	Ensure directory tree exists
resolve()	Canonical absolute path
read_bytes() / write_bytes()	Binary without encoding concerns
glob("**/*.py")	Recursive pattern match
rglob("*.log")	Shorthand for glob("**/*.log")
with_suffix(".bak")	Change extension safely

---

Buffering and performance

from pathlib import Path

path = Path("large.log")

# Unbuffered binary — immediate OS writes
with path.open("wb", buffering=0) as f:
    f.write(b"urgent")

# Line buffering — text mode flushes on newline
with path.open("w", encoding="utf-8", buffering=1) as f:
    f.write("line\n")

# Default block buffering — 8 KiB typical
with path.open("rb") as f:
    while chunk := f.read(65536):  # 64 KiB chunks
        process(chunk)

Strategy	Memory	When to use
read()	Entire file	Small files (< few MB)
Line iteration	One line at a time	Log parsing, text pipelines
Fixed-size chunks	Constant buffer	Binary streams, ETL
mmap	OS page cache	Random access in huge files

Always set encoding for text

Same rule as intermediate I/O: encoding="utf-8" avoids platform-dependent defaults. See File Handling.

---

Memory-mapped files (mmap)

import mmap
from pathlib import Path

path = Path("huge.bin")
with path.open("rb") as f:
    with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
        # Random access without loading entire file into Python heap
        header = mm[0:16]
        index = mm.find(b"\x00\x01")

Memory maps let the OS manage paging — efficient for read-heavy random access on files larger than RAM. Writes require ACCESS_WRITE and care with concurrent access.

---

Atomic and safe writes

Crash mid-write corrupts data if you write directly to the target path. Pattern: write temp → fsync → rename.

import os
import tempfile
from pathlib import Path

def atomic_write(path: Path, data: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    fd, tmp_name = tempfile.mkstemp(dir=path.parent, text=True)
    tmp_path = Path(tmp_name)
    try:
        with os.fdopen(fd, "w", encoding="utf-8") as f:
            f.write(data)
            f.flush()
            os.fsync(f.fileno())
        tmp_path.replace(path)
    except BaseException:
        tmp_path.unlink(missing_ok=True)
        raise

On POSIX, rename over an existing file is atomic on the same filesystem. Cross-device moves need shutil.move (not atomic).

---

tempfile module

import tempfile
from pathlib import Path

with tempfile.TemporaryDirectory() as tmpdir:
    work = Path(tmpdir) / "scratch.txt"
    work.write_text("temp data", encoding="utf-8")
# Directory and contents deleted on context exit

with tempfile.NamedTemporaryFile("w+", encoding="utf-8", delete=False) as f:
    f.write("persist briefly")
    name = f.name

API	Behavior
TemporaryDirectory()	Auto-cleaned dir (context manager)
NamedTemporaryFile(delete=False)	Survives after close — clean up manually
mkstemp()	Low-level fd + path; you own cleanup
gettempdir()	System temp location

---

shutil — high-level file operations

import shutil
from pathlib import Path

src = Path("project/")
dst = Path("backup/project/")

shutil.copy2(src / "config.yaml", dst / "config.yaml")  # preserves metadata
shutil.copytree(src / "data", dst / "data", dirs_exist_ok=True)  # 3.8+
shutil.move(str(src / "old.log"), str(dst / "archive.log"))
shutil.disk_usage("/")  # (total, used, free)

import shutil

shutil.rmtree(path)  # recursive delete — no undo

Interview trap — shutil.copy vs copy2

copy copies content only; copy2 also copies metadata (mtime, mode). Neither copies extended attributes on all platforms.

---

os and low-level I/O

import os
from pathlib import Path

path = Path("data.bin")
fd = os.open(path, os.O_RDWR | os.O_CREAT, mode=0o644)
try:
    os.write(fd, b"binary")
    os.lseek(fd, 0, os.SEEK_SET)
    data = os.read(fd, 1024)
finally:
    os.close(fd)

Prefer pathlib unless you need fine-grained control (e.g., os.open flags, sendfile, pipe).

---

io module — in-memory streams

import io

# Treat string as file — useful for testing parsers
buf = io.StringIO("line1\nline2\n")
assert buf.readline() == "line1\n"

binary = io.BytesIO(b"\x00\x01\x02")
binary.seek(0)

Use StringIO/BytesIO in Unit Testing to simulate file input without touching disk.

---

Context managers and contextlib

from contextlib import contextmanager
from pathlib import Path

@contextmanager
def open_locked(path: Path, mode: str = "r"):
    path = Path(path)
    lock = path.with_suffix(path.suffix + ".lock")
    lock.touch(exist_ok=False)
    try:
        with path.open(mode, encoding="utf-8") as f:
            yield f
    finally:
        lock.unlink(missing_ok=True)

Stack multiple managers:

from contextlib import ExitStack

with ExitStack() as stack:
    files = [stack.enter_context(p.open()) for p in paths]
    # all files closed on exit

---

Working with large text and CSV

import csv
from pathlib import Path

path = Path("huge.csv")
with path.open(newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    for row in reader:
        handle(row)  # one row in memory at a time

For JSON lines (NDJSON):

import json
from pathlib import Path

with Path("events.ndjson").open(encoding="utf-8") as f:
    for line in f:
        event = json.loads(line)

---

File descriptors and with pitfalls

f = open("out.txt", "w", encoding="utf-8")
f.write("data")
# forgot f.close() — leak until GC (non-deterministic)

# Correct
with open("out.txt", "w", encoding="utf-8") as f:
    f.write("data")

On CPython, file objects are promptly closed when the with block exits, even on exceptions — see Error Handling.

---

Permissions and security

import os
from pathlib import Path

path = Path("secret.key")
path.write_text("key-material", encoding="utf-8")
os.chmod(path, 0o600)  # owner read/write only (POSIX)

Never world-read credential files. Prefer environment variables or a secrets manager — Managing Secrets.

---

Async file I/O note

Standard pathlib and open() are blocking. In asyncio apps, use asyncio.to_thread() for disk I/O or libraries like aiofiles to avoid blocking the event loop. Pair with HTTP Requests for async download pipelines.

---

Interview traps (quick reference)

Trap	What goes wrong	Safe approach
read() on multi-GB file	MemoryError	Chunked reads or mmap
Writing directly to production path	Corrupt file on crash	Temp + atomic rename
CSV without newline=""	Extra blank rows	Follow csv module requirement
shutil.rmtree without confirmation	Irrecoverable data loss	Guard with checks / dry-run
Relative paths + wrong cwd	FileNotFoundError	Anchor with __file__ or CLI args
Assuming close() in __del__	Nondeterministic cleanup	Always use with
encoding omitted on Windows	Mojibake	encoding="utf-8"

Mental model checklist

1. When should you use mmap instead of read()?
2. What makes a write "atomic" on POSIX?
3. What does newline="" do for CSV on Windows?
4. How does TemporaryDirectory clean up resources?
5. What is the difference between copy and copy2?
6. Why use chunked reads for binary files?

What's next

Topic	Page
Foundations	File Handling
I/O errors	Error Handling
Secret files	Managing Secrets
Testing parsers	Unit Testing