<aside> 🧭

Module 10 · I/O, the block layer and iowait

Every slow thing in the last two modules ended at a disk. This module opens that up: what happens between a write() and the storage device, why iowait is one of the most misleading numbers in Linux, and how to tell a disk that is genuinely saturated from one that is merely busy.

🧠 concept → 🧩 real-world analogy → 🧪 exercise → ✅ expected result (hidden) → 🎯 interview questions (hidden)

</aside>

<aside> ✅

Before you start, you should already know:

From Module 03 — file descriptors, inodes, filesystems and mounts.

From Module 07 — the us/sy/wa split in top and vmstat, that wa is a kind of idle, and how to read /proc/pressure/.

From Module 08 — major page faults, and that a major fault means waiting on storage.

From Module 09 — the page cache, dirty pages, writeback, and fsync. This module is the other side of all of them.

</aside>

🪢 Part A · From write() to the device

A1 · The layers, and what each one is for

<aside> 📖

Official docs: Block layer — kernel docs · write(2) · read(2)

</aside>

When a program calls write(), a surprising number of things happen before anything reaches a disk — and on most calls, nothing reaches a disk at all. Knowing the layers matters because each one has its own metrics, and measuring at the wrong layer is how most disk investigations go wrong.

flowchart TD
    A["write() in your program"] --> B["VFS<br>which filesystem, which inode"]
    B --> C{"Page cache"}
    C -->|"write"| D["Mark pages dirty<br>return to the program NOW"]
    C -->|"read hit"| E["Copy from RAM<br>no device involved"]
    D --> F["Writeback later<br>or on fsync"]
    C -->|"read miss"| F
    F --> G["Filesystem<br>file offset to device blocks"]
    G --> H["Block layer<br>queue, merge, order"]
    H --> I["Driver"]
    I --> J["Device"]

<aside> 💡

Set this block to Preview using the ••• menu on its right to see the diagram instead of the code. Notion does not do that automatically.

</aside>

<aside> 🧠

The counter-intuitive part. The two arrows that matter most in that diagram are the ones that stop early. A read that hits the page cache never reaches the block layer, and a write returns to your program the moment the page is marked dirty. So on a healthy machine, most I/O calls involve no I/O. Everything below the page cache is the exception path — which is exactly why per-device metrics and application-level metrics so often disagree.

</aside>

<aside> 🧩

Real-world analogy — posting a parcel

You hand a parcel to the office post room and walk away. From your point of view the parcel is sent. That is write() returning.

The post room holds it in a tray, and periodically a van takes a load to the depot. The van does not leave for each parcel — it waits until it has a load, and it groups parcels going to the same place. That is writeback and request merging.

The depot sorts by destination and decides what goes on which lorry in what order. That is the block layer and its scheduler.

And only then does anything actually travel.

Now the important part: if someone asks "how fast is our post?", the answer depends entirely on where they are standing. In your office it is instant. At the depot it is hours. Measuring at the wrong point gives a confident, precise, useless number — which is what happens when a team benchmarks storage with dd and no fsync.

Where the analogy stops working. You could walk down and check on your parcel. A program has no visibility below the syscall it made, which is why these layers have to expose their own counters.

</aside>

🧪 Exercise A1.1 — Watch a read stop at each layer

# We need a file bigger than a trivial cache hit
dd if=/dev/zero of=/tmp/layers bs=1M count=256 status=none
sync

# Counters for the device holding /tmp, before and after each read.
# Field 3 of /proc/diskstats is the device name, field 6 is sectors read.
SRC=$(findmnt -no SOURCE --target /tmp | sed 's/\[.*\]//')
DEV=$(lsblk -no PKNAME "$SRC" 2>/dev/null | head -1)
[ -z "$DEV" ] && DEV=$(lsblk -no KNAME "$SRC" 2>/dev/null | head -1)
echo "device: $DEV"
sectors() { awk -v d="$DEV" '$3==d {print $6}' /proc/diskstats; }

# --- cold read: this must reach the device ---
sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
BEFORE=$(sectors)
cat /tmp/layers > /dev/null
AFTER=$(sectors)
echo "cold read moved $(( (AFTER - BEFORE) / 2 )) KiB from the device"

# --- warm read: identical command ---
BEFORE=$(sectors)
cat /tmp/layers > /dev/null
AFTER=$(sectors)
echo "warm read moved $(( (AFTER - BEFORE) / 2 )) KiB from the device"

# --- and how many syscalls each one made ---
strace -c -f -e trace=read,write cat /tmp/layers > /dev/null 2>/tmp/sc
tail -6 /tmp/sc

rm -f /tmp/layers /tmp/sc

• ✅ Expected result — click to reveal

<aside> 🏭

Now imagine this at 500 hosts. This is why "the application says storage is slow but the storage team says the volume is idle" is such a common standoff, and both are usually telling the truth. The application is measuring syscall latency, which includes page-cache misses, writeback stalls, fsync waits and filesystem journal contention. The storage team is measuring the device. Collect both, and collect the layer in between — /proc/diskstats deltas and /proc/pressure/io — or every incident becomes an argument.

</aside>

A2 · The block layer — queueing, merging, and blk-mq

<aside> 📖

Official docs: Multi-Queue Block IO Queueing Mechanism (blk-mq) · sysfs-block ABI · lsblk(8)

</aside>