<aside> 📈

Module 14 — Logs, Troubleshooting, and Performance

This is where the whole track converges: the module about incidents. "The server is slow," "something's broken," "it worked yesterday" — this teaches the systematic method that turns panic into diagnosis. You'll learn to read a machine's vital signs, find the story in its logs, and follow a repeatable troubleshooting method instead of guessing. Nearly every senior-level Linux interview is, at heart, "here's a broken server — what do you do?"

Legend used throughout: 🧠 concept → 🧩 analogy → 🧪 exercise → ✅ expected result (hidden) → 🎯 interview questions (hidden)

</aside>

<aside> 🚦

Before you start

You need the whole track so far, Modules Module 1 — What Linux Is–Module 13 — Networking — this module composes everything: processes (8), services and journalctl (11), storage (12), networking (13). Load-bearing especially: top/ps/states (Module 8), the journal (Module 11), df/du/free (Module 12). Tools: an Ubuntu 24.04 machine; we install sysstat (for iostat) in Part C. Everything here is read-only inspection — safe to run anywhere, which is the point: diagnosis observes before it acts.

Goes deeper: the kernel mechanics behind these signals — scheduling and load, virtual memory and the OOM killer, the I/O path and iowait, and a formal performance methodology — are four companion modules in the Operating Systems track (OS 07–10 and 14). This module is the hands-on operator's view; those are the theory.

</aside>

<aside> ℹ️

This page uses Mermaid diagram blocks. Notion shows them as code by default — click the block and set it to Preview to see the diagram. This reminder appears once per page.

</aside>

Part A — Reading the vital signs

A1. uptime and load average

<aside> 📖

Official docs: uptime(1)

</aside>

🧠 uptime gives a one-line health summary: how long the machine's been up, how many users, and the load average — three numbers that are the single most-quoted, most-misunderstood metric in Linux. Load average is the average number of processes wanting to run (running, or waiting to run, plus on Linux those in uninterruptible I/O wait — Module 8's D state) over the last 1, 5, and 15 minutes. The three timescales show trend: 2.0, 1.0, 0.5 means load is rising (recent higher than older); 0.5, 1.0, 2.0 means it's falling. Reading the trend, not just the instant, is the skill.

<aside> 🧠

Counter-intuitive: a load average of 4 is a disaster on one machine and totally idle on another — it's meaningless without the core count. Load is a count of processes wanting CPU; whether that's "too much" depends entirely on how many CPUs you have to run them. Load 4 on a 1-core machine means work is queuing 4-deep — badly overloaded. Load 4 on a 4-core machine means every core is busy with nothing waiting — perfectly, fully utilized. Load 4 on a 16-core machine is 75% idle. So the rule is load ÷ cores: near 1.0 = fully used, well above 1.0 = overloaded (work is queuing), well below = idle. Always pair uptime with nproc (Module 8) — a load number without the core count is a number without units. This single misunderstanding causes more false alarms and missed real problems than any other metric.

</aside>

<aside> 🧩

Real-world analogy — the checkout queue

Load average is the number of shoppers wanting to check out. But "8 people waiting" only means something once you know how many tills are open: 8 shoppers across 8 tills is one each — smooth. 8 shoppers at 1 till is a long, angry queue. The three numbers (1/5/15 min) are whether the queue is growing (a rush starting) or clearing (winding down).

Where the analogy stops working. Shoppers only want the till — they're all "runnable." Linux load also counts shoppers stuck waiting for a price check (uninterruptible I/O wait, D state) who aren't using a till at all. So a high Linux load can mean "CPU overloaded" or "everything's stuck waiting on a slow disk" — two completely different problems with the same load number. Distinguishing them (is it us/sy CPU, or wa iowait?) is what top and vmstat are for, and why load alone never finishes a diagnosis.

</aside>

🧪 Exercise A1.1 — Read the load, in context (read-only, safe anywhere)

uptime                   # the one-line summary with the three load numbers
nproc                    # how many cores? (load is meaningless without this)
cat /proc/loadavg        # the kernel's raw load figures (what uptime reads)

• ✅ Expected result — click to reveal

A2. top — the live triage screen

<aside> 📖

Official docs: top(1)

</aside>

🧠 Module 8 introduced top as a process viewer; here it's the incident triage screen — the "server is slow, what first?" answer. Its header answers the four triage questions in order: load (A1, first line), the task census (running/sleeping/zombie — Module 8), the %Cpu(s) line (where the CPU's time goes), and the memory lines. Then the process table, sortable by CPU (P) or memory (M), names the culprit. The single most diagnostic field is the %Cpu(s) breakdown, because it tells you what kind of slow you have — and each kind sends you somewhere different.

The %Cpu(s) letters, and what each high value means:

<aside> 🧩

Real-world analogy — the hospital triage nurse

top is the triage nurse who, in ten seconds, reads the vital signs and routes you: chest pain here, broken bone there. The %Cpu(s) line is those vitals — it doesn't treat anything, but it sorts the problem into the right category (CPU-bound? I/O-bound? idle-but-slow?), which is 90% of solving it fast.

Where the analogy stops working. A triage nurse sees the patient in front of them. top shows only this instant, refreshed and discarded — the spike that caused last night's page is simply not there this morning (which is why history-keeping monitoring exists, A-part orange callout). And top can mislead: a process can be the victim rather than the cause (starved by something else), so the nurse's first read is a hypothesis to confirm, not a verdict.

</aside>

🧪 Exercise A2.1 — Triage in ten seconds (read-only, safe anywhere)