<aside> 🧭
Module 14 · Performance methodology and tracing
Modules 07 to 13 each handed you a set of numbers: run queues, page faults, iowait, cgroup pressure, capability masks. This module is about the part nobody teaches — which number to look at first, and how to get from "the service is slow" to a named cause without guessing.
🧠 concept → 🧩 real-world analogy → 🧪 exercise → ✅ expected result (hidden) → 🎯 interview questions (hidden)
</aside>
<aside> ✅
Before you start, you should already know:
From Module 07 — run queue, load average including D-state, and that iowait is a subset of idle.
From Module 04 — signal masks in /proc/<pid>/status.
From Module 08 — minor and major page faults.
From Module 09 — page cache, MemAvailable, and PSI.
From Module 10 — iostat -x, and that %util is meaningless on a parallel device.
From Module 12 — cgroups, and cpu.stat's nr_throttled.
From Module 13 — capabilities, seccomp, and that --privileged turns the syscall filter off.
Tools used here: sysstat (pidstat, mpstat, iostat, sar), procps (vmstat, free, uptime), strace, and perf. On Debian/Ubuntu: sudo apt-get install -y sysstat strace linux-tools-common linux-tools-generic. On RHEL-family: sudo dnf install -y sysstat strace perf.
</aside>
top first" is the wrong instinct<aside> 📖
Official docs: The USE Method · Linux Performance Analysis in 60,000 Milliseconds · Monitoring Distributed Systems (Google SRE book)
</aside>
Everyone has a first command. top, or htop, or whatever their last team used. It is not a bad command — but starting there means you are not investigating, you are checking your favourite place, and the outcome depends on whether the problem happens to live there.
Brendan Gregg gave these habits names, and recognising your own in the list is the first useful step:
| Anti-method | What it looks like | Why it fails |
|---|---|---|
| Streetlight | You run the tools you know, because you know them | The drunk looking for keys under the streetlight, because the light is better there. It finds problems only where your tools already point |
| Random change | Change a setting, redeploy, see if it helped | You cannot tell a fix from a coincidence, and every change is now permanent because nobody dares revert it |
| Blame someone else | "It's the network." "It's the database." | Costs another team a day, and produces no evidence either way |
| Tool-first | "Let's get a flame graph." | Sometimes right by luck. A flame graph of a process that is blocked on disk shows you nothing at all |
A method replaces all of them with the same idea: enumerate everything that could be the answer, then eliminate. You start from a list, not from a tool, and the list is the thing worth memorising.
<aside> 🧠
The counter-intuitive part. Performance work feels like it should reward knowing the most tools. It does not. It rewards having a complete list of suspects, because the failure mode that costs you the afternoon is never "I did not know that command" — it is "I never thought to look there." Two engineers with the same tools and different methods produce wildly different results, and the one with the method is usually the slower typist.
</aside>
<aside> 🧩
Real-world analogy — the car that will not start
Two mechanics. The first opens the bonnet and checks the spark plugs, because the last three cars that would not start had bad plugs. If it is the plugs, they are a hero in four minutes. If it is not, they are now looking at the alternator, then the starter motor, in whatever order occurs to them, and at some point they will start replacing parts to see what happens.
The second reaches for a checklist that came with the car: does it crank? Is there fuel pressure? Is there spark? Is there compression? Four questions, and every possible cause of "will not start" is inside one of them. They may not be faster on the plug-fouling case. They are dramatically faster on everything else, and — the part that matters at three in the morning — they can tell you when they are finished, because the list is finite.
The first mechanic's real problem is not ignorance. They may know more about engines than the second. Their problem is that they have no way to know what they have not checked.
Where the analogy stops working. A car has one fault at a time and a fixed parts list. A distributed system routinely has two interacting causes, and the list of resources grows every time someone adds a service — which is why the methods below are framed around categories of resource rather than specific components.
</aside>
🧪 Exercise A1.1 — Ask the machine three questions in ten seconds
Before reading Section A2, run this on a machine doing something. Time yourself.
# Put two CPU burners on the box so there is something to see
for i in 1 2; do ( timeout 15 bash -c 'x=0; while :; do x=$((x+1)); done' & ); done
sleep 3
# Question 1: is this getting worse, or has it always been like this?
uptime
# Question 2: where is the time going, machine-wide?
vmstat 1 3
# Question 3: is it every CPU, or one hot CPU?
mpstat -P ALL 1 2 | tail -6
<aside> 🏭
Now imagine this at 500 hosts. The reason to learn a fixed checklist is that it is the only thing that survives being handed to someone else. A senior engineer's intuition does not fit in a runbook and does not work at 3 a.m. on the tenth incident of the week. Ten commands with a written interpretation of each do. Put the checklist in the runbook, and put its output in the incident template so the first responder collects it before they start theorising.
</aside>