eBPF Performance Monitoring with Aya
Performance problems in production systems are usually a puzzle with missing pieces. You know the symptom — high latency, unexpected CPU usage, I/O spikes — but the signals you have don’t point to the cause. The missing pieces are often things like cache misses starving the pipeline, NUMA traffic slowing down memory access, or a scheduler that can’t find a runnable task fast enough.
This tutorial is about filling in those pieces. We’re going to instrument a Linux system to capture a detailed performance picture across three layers:
Hardware — Performance Monitoring Counters (PMCs) that live on the CPU chip itself. These count events like cache line loads, branch mispredictions, and TLB walks. They’re the closest thing to a direct line into what the CPU is actually doing.
Kernel — eBPF programs attached to kprobes and tracepoints let us observe kernel behavior without adding latency. We can watch how tasks get scheduled, how I/O requests enter the block layer, and how vhost/virtio rings get driven.
Procfs and sysfs — The kernel exposes a huge amount of state through virtual filesystems. NUMA hit/miss statistics, thermal zone temperatures, uncore IMC counters, per-CPU stats — all readable from userspace with no eBPF required.
The tool that ties it all together is Aya — a Rust library for building eBPF programs without relying on LLVM/BCC/libbpf. We’ll write eBPF programs in Rust, and use the same Rust codebase to read userspace sources and aggregate everything into a coherent output.
The Metrics
Here’s what we’ll capture and why:
CPU Internals
- L1/L2/L3 cache miss rates — When a core can’t find data in cache, it stalls. High L3 miss rates often point to poor data locality or memory bandwidth saturation.
- Branch mispredict rate — Modern CPUs speculatively execute. A mispredict throws away work. High mispredict rates point to unpredictable control flow or hot loops.
- IPC and stall ratio — Instructions per cycle (IPC) measures actual throughput. Stall ratio is the fraction of cycles the core was waiting, not executing.
- dTLB/iTLB miss — The Translation Lookaside Buffer caches virtual-to-physical address translations. A miss means a costly page table walk.
Memory and NUMA
- Uncore IMC bandwidth — The Integrated Memory Controller sits off-chip. It has its own performance counters. Saturation means memory bandwidth is the bottleneck.
- NUMA remote ratio — On a multi-socket system, accessing memory attached to a remote socket is slower. The remote ratio tells you what fraction of memory access is remote.
- Page migration rate — NUMA balancing moves pages between nodes. High migration rates mean the system is fighting itself.
- Hugepage utilization — Hugepages reduce TLB pressure. If the pool is exhausted or transparent hugepages aren’t coalescing, applications aren’t getting the benefit they should.
I/O
- IOPS pattern entropy — Predictable I/O patterns are easier for the kernel to batch. High entropy means the I/O is random and may saturate queues.
- vhost queue depth p50/p99 — How many requests are buffered in the vhost/virtio ring. High p99 means occasional stalls.
- virtio ring stalls — How often the virtio ring can’t proceed because it’s waiting for descriptors.
Scheduler
- runqueue wait p50/p99 — How long a task sits waiting on the runqueue before it gets a CPU. High p99 means scheduling latency spikes.
- vCPU steal time — On virtualized systems, “steal” is time the guest wanted to run but the hypervisor didn’t give it. High steal means the host is oversubscribed.
- involuntary ctxsw rate — How often tasks get kicked off the CPU involuntarily (preempted, time slice expired). High rates can mean too many CPU-bound tasks.
Thermal
- per-core thermal headroom — The gap between current temperature and the thermal throttle point. Headroom means performance isn’t thermally constrained.
The tutorial:
- Parts 1–2: Architecture and project setup
- Parts 3–4: Hardware PMCs with
perf_event_open - Parts 5–6: Cache and TLB metrics; scheduler tracing with eBPF
- Parts 7–8: NUMA and memory metrics; uncore IMC bandwidth
- Parts 9–10: Thermal monitoring; block I/O tracing and entropy
- Parts 11–12: vhost and virtio ring instrumentation; histograms for queue depth
- Part 13: Wiring all three sources into one Tokio event loop
Next: Part 1 — Three Sources of Signal — Where our data comes from: hardware PMCs, kernel tracepoints, and virtual filesystems.
Part 1 — Three Sources of Signal
Performance data comes from three places on Linux. Understanding which source a metric comes from tells you a lot about how to capture it, what it means, and where its limits are.
Source 1: Hardware Performance Counters (PMCs)
Modern CPUs have dedicated hardware counters on-die. They’re called PMCs — Performance Monitoring Counters. They count things like:
- How many times a cache line was requested and missed
- How many branches executed and how many were mispredicted
- How many instructions retired and how many cycles were stalled
- How many TLB (Translation Lookaside Buffer) walks happened and how many missed the hardware TLB
PMCs are read through the perf_event_open syscall. On x86, the underlying hardware is the Performance Counters MSRs (Model-Specific Registers). On ARM, there’s the ARM PMU (Performance Monitoring Unit). The syscall abstracts this, but the available events depend on your CPU microarchitecture.
The kernel exposes a curated list of events through /dev/cpu/*/msr (requires root) and through perf list (user-accessible for most events). If you only need the universal hardware events (instructions, cycles, cache references), the perf-event crate wraps perf_event_open with a safe Rust interface. We don’t use it here because we need raw PMC events (cache misses, TLB walks — the hardware lookup triggered by a TLB miss, branch mispredicts) that vary by CPU microarchitecture — the crate doesn’t expose PERF_TYPE_RAW on all platforms. Part 3 shows the hand-rolled struct and the direct syscall.
Constraints:
- Most PMCs require root or
CAP_SYS_ADMIN - Not all events are available on all CPUs
- On hypervisors, some events may not reflect guest-observed behavior accurately
- Counting across multiple CPUs requires either per-CPU file descriptors or multiplexing
Source 2: Kernel Tracepoints and Kprobes
The Linux kernel emits events at interesting points in its execution. These come in two forms:
Tracepoints are stable hooks placed in the kernel by developers. They have stable names and argument formats. Examples:
sched:sched_waking— a task is about to be wokensched:sched_switch— the scheduler switched from one task to anotherblock:block_bio_queue— a block I/O request was submittedirq:softirq_entry— a softirq started executing
Kprobes are dynamic probes that can be placed at almost any kernel function entry or return. They’re less stable (function names change between kernel versions) but much more powerful — you can probe any function, not just the ones with tracepoints.
Both tracepoints and kprobes are programmable via eBPF. This is the core of what Aya lets us do in Rust — write eBPF programs that read data from these hooks and push it into maps that userspace can read. Part 6 shows the scheduler tracing; Part 10 shows block I/O tracing; Part 11 shows vhost/virtio instrumentation.
Key maps:
BPF_MAP_TYPE_PERF_EVENT_ARRAY— ring buffer for sending structured events to userspaceBPF_MAP_TYPE_HASH— key-value store for counters and stateBPF_MAP_TYPE_ARRAY— indexed array, good for histogramsBPF_MAP_TYPE_RINGBUF— lock-free ring buffer, newer and faster than perf events
Constraints:
- Kprobe function names are kernel-version specific
- eBPF programs are verified — you can’t write to arbitrary memory or loop unboundedly
- The eBPF VM has a 512-byte stack limit (no heap)
- CO-RE (Compile Once, Run Everywhere) with BTF (BPF Type Format — the kernel’s type information, exposed as structured data) makes kprobes portable; without it, you need kernel headers for each target version
Source 3: Procfs and Sysfs
The kernel exposes a huge amount of state through two virtual filesystems that don’t exist on disk:
/proc/ — process and system information. Relevant files:
/proc/cpuinfo— CPU model, microarchitecture, flags/proc/vmstat— virtual memory statistics, including NUMA page stats/proc/schedstat— scheduler statistics per CPU/proc/interrupts— interrupt counts per CPU/proc/loadavg— load average
/sys/ — kernel data structures organized as a tree. Relevant paths:
/sys/devices/system/cpu/— per-CPU attributes/sys/class/thermal/thermal_zone*/— thermal zones with current temperature/sys/bus/event_source/devices/— available perf events/sys/kernel/mm/— hugepages and transparent hugepage settings/sys/class/block/— per-block-device statistics/sys/devices/system/node/— NUMA node memory statistics
These are readable with standard file I/O — no root required for most files, and no eBPF required.
Constraints:
- Values may be stale — the kernel updates these files at its own pace, not on every event
- No subscription mechanism — you poll them, you don’t get notified
- No historical data — you see the current value or a cumulative counter, not a time series
- File paths and formats vary between kernel versions (though the ones we use are stable across 5.x and 6.x)
The Hybrid Architecture
Our monitoring system uses all three sources together. The architecture looks like this:
┌──────────────────────────────────────────────────┐
│ Userspace (monitor) │
│ │
│ ┌─────────────┐ ┌──────────────┐ ┌────────┐ │
│ │ perf_event │ │ ring buffer │ │procfs/ │ │
│ │ open poll │ │ reader │ │sysfs │ │
│ └──────┬──────┘ └──────┬───────┘ └───┬────┘ │
│ │ │ │ │
└─────────┼────────────────┼──────────────┼───────┘
│ │ │
perf_event_open() eBPF maps file I/O
▲ ▲ ▲
│ │ │
┌─────────┴────────────────┼──────────────┼───────┐
│ Linux Kernel │ │ │
│ │ │ │
│ ┌─────────────────┐ │ │ │
│ │ PMC counters │ │ │ │
│ │ (perf_event_open│ │ │ │
│ │ file desc.) │ │ │ │
│ └─────────────────┘ │ │ │
│ read via fd │ │ │
│ │ │ │
│ ┌─────────────────────────────────┐ │ │
│ │ eBPF programs │ │ │
│ │ ┌──────────────────────────┐ │ │ │
│ │ │ scheduler tracepoints │ │ │ │
│ │ │ (waking, switch) │ │ │ │
│ │ └──────────────────────────┘ │ │ │
│ │ ┌──────────────────────────┐ │ │ │
│ │ │ block I/O, vhost, etc. │ │ │ │
│ │ └──────────────────────────┘ │ │ │
│ │ writes to eBPF maps ──────────┘ │ │
│ └─────────────────────────────────┘ │
│ │
│ ┌─────────────┐ ┌────────────────┐ │
│ │ /proc/vmstat│ │/sys/class/ │ │
│ │ /proc/stat │ │thermal/ │ │
│ └─────────────┘ └────────────────┘ │
└─────────────────────────────────────────────────┘
PMC counters are not eBPF programs — they’re file descriptors opened via perf_event_open and read directly. eBPF programs are the ones attached to kernel tracepoints and kprobes. They write aggregated data (histograms, counters) into eBPF maps, which the ring buffer reader in userspace consumes. Procfs and sysfs are plain file reads — no special API, no eBPF.
The user-space program polls all three sources in a single event loop. The eBPF programs handle the in-kernel aggregation so we only get summaries over the ring buffer rather than a firehose of raw events.
What Aya Provides
Aya is the Rust library that ties the eBPF and userspace halves together. It handles:
- Compiling eBPF programs via
aya-build(runs automatically inbuild.rs— no separate build step) - Loading and attaching programs from Rust
- Creating and populating maps (ring buffers, hashes, histograms)
- Reading from maps in the user-space half
Aya is unusual among eBPF libraries because it doesn’t depend on libbpf, BCC, or a C toolchain. Everything is Rust, end to end.
Next: Part 2 — Project Setup and Minimal eBPF — Scaffold the Aya project, write a tracepoint handler, and read scheduler events in userspace.
Part 2 — Project Setup and Minimal eBPF
Before we write any instrumentation, we need a working project that compiles and runs. Aya projects have two halves: the user-space Rust program and the eBPF programs. Getting the build tooling right is the first thing people get stuck on, so let’s do it carefully.
Prerequisites
You’ll need:
# Rust stable and nightly (needed for eBPF compilation)
rustup install stable
rustup toolchain install nightly --component rust-src
# Add the eBPF target to the nightly toolchain
# (Aya compiles eBPF programs as a separate target)
rustup target add bpfel-unknown-none --toolchain nightly
# bpf-linker: compiles eBPF bytecode from Rust
cargo install bpf-linker
# bpftool: generates Rust bindings from BTF info
# On Ubuntu, install from your package manager first, or build from source:
# https://github.com/libbpf/bpftool
sudo apt install linux-tools-$(uname -r)
# cargo-generate: scaffolds the Aya template
cargo install cargo-generate
Check your kernel version — eBPF is generally well-supported on kernels 5.8+, but some features (ringbuf, BTF) work better on 5.10+:
uname -r
⚠️ One Version Trap to Watch For
Aya has two separate crate families with independent version tracks: aya (user-space) and aya-ebpf (eBPF kernel programs). They don’t share a version number. When you see aya = "0.13", the companion eBPF crate might be 0.1, 0.2, or something else entirely — check crates.io to confirm the current version.
Some Aya versions have been yanked from crates.io due to build issues. As of June 2026:
| Crate | Latest | Tested with (this tutorial) | Yanked versions |
|---|---|---|---|
aya | 0.14.0 | 0.13.1 | 0.13.2 |
aya-ebpf | 0.2.0 | 0.1.1 | 0.1.2 |
aya-log-ebpf | 0.2.0 | 0.1.0 | 0.1.1 |
aya-log | 0.3.0 | 0.2.1 | — |
aya-build | 0.2.0 | 0.1.3 | — |
The 0.14.0 release (June 2026) bumps the aya user-space crate and the aya-ebpf/aya-log-ebpf eBPF crates to new minor versions. The code in this tutorial was written and validated against 0.13.x — the Cargo.toml versions below pin to the 0.13.x line. If you want to use 0.14.0+, check the Aya changelog for breaking changes; the APIs shown here may differ.
Cargo automatically skips yanked versions when resolving semver ranges — aya = "0.13" resolves to 0.13.1, not the yanked 0.13.2. The version specifications below use semver ranges that pin to the 0.13.x line. If you pin to an exact version, avoid the ones in the yanked column.
If cargo update pulls in mismatched versions across the two crate families, pin them explicitly in Cargo.toml.
Scaffolding the Project
The Aya team provides a template. Use it with the program type and tracepoint details pre-filled — otherwise cargo generate will prompt you interactively, which breaks the copy-paste flow:
cargo generate --name perf-monitor \
-d program_type=tracepoint \
-d tracepoint_category=sched \
-d tracepoint_name=sched_switch \
https://github.com/aya-rs/aya-template
What each argument does:
--name perf-monitor— the name of the directory and the Rust workspace. This becomes the workspace root, and the three crates get named accordingly.-d program_type=tracepoint— tells the template to generate a tracepoint program. The template supports many types (xdp,kprobe,uprobe,tracepoint, etc.). Each type changes the generated code: a tracepoint program reads from aTracePointContext, an XDP program reads from anXdpContext, and so on. We picktracepointbecause our first instrument targets a kernel tracepoint.-d tracepoint_category=sched— the tracepoint category (also called the subsystem). Kernel tracepoints are organized ascategory:name. Theschedcategory contains scheduler events:sched_switch,sched_wakeup,sched_waking, etc.-d tracepoint_name=sched_switch— the specific tracepoint event.sched_switchfires every time the kernel switches from one task to another. It’s the most fundamental scheduler tracepoint — it tells you what ran and when.https://github.com/aya-rs/aya-template— the template repository.cargo generatedownloads this, replaces placeholders with the values you passed (-dflags), and writes the result to a new directory.
The -d flags are how you answer the template’s questions ahead of time. Without them, cargo generate prompts you interactively.
Important: The template uses git dependencies by default. After generating, switch them to crates.io versions for stability. Edit Cargo.toml in the workspace root and perf-monitor-ebpf/Cargo.toml:
# Before (git — moves, may break)
aya = { git = "https://github.com/aya-rs/aya" }
# After (crates.io — stable, tested)
aya = "0.13" # resolves to 0.13.1 (0.13.2 was yanked)
aya-build = "0.1"
aya-ebpf = "0.1"
aya-log = "0.2"
aya-log-ebpf = "0.1"
Then run cargo update to resolve.
This creates a workspace with three crates:
perf-monitor/ ← workspace root (Cargo.toml at root)
├── perf-monitor/ ← user-space program (what we write)
│ ├── Cargo.toml
│ ├── build.rs
│ └── src/main.rs
├── perf-monitor-ebpf/ ← eBPF programs (compiled to BPF bytecode)
│ ├── Cargo.toml
│ ├── build.rs
│ └── src/main.rs
└── perf-monitor-common/ ← code shared between userspace and eBPF
├── Cargo.toml
└── src/lib.rs
The eBPF build is handled automatically: perf-monitor-ebpf/build.rs runs the Aya build toolchain, which compiles the eBPF programs to BPF bytecode and embeds them into OUT_DIR. One cargo build compiles both halves.
The tracepoint template scaffolds a working program already wired to sched:sched_switch. We’ll replace the generated body with our own code, but the structure — the workspace, the three crates, the build configuration — is what we need.
The Two Halves
eBPF Programs (perf-monitor-ebpf/)
The eBPF programs are written in Rust but compiled to BPF bytecode by the build.rs script. The Aya aya-ebpf crate provides the Rust API for eBPF maps, programs, and context objects — no standard library, no heap, strict verifier.
A simple tracepoint program looks like this:
#![allow(unused)]
fn main() {
// perf-monitor-ebpf/src/main.rs
#![no_std]
#![no_main]
use aya_ebpf::programs::TracePointContext;
use aya_ebpf::macros::tracepoint;
use aya_ebpf::maps::RingBuf;
use aya_ebpf::helpers::{bpf_ktime_get_ns, bpf_get_smp_processor_id};
// Events we emit to userspace
#[derive(Clone, Copy)]
#[repr(C)]
struct SchedulerEvent {
cpu_id: u32,
prev_pid: u32,
next_pid: u32,
timestamp: u64,
}
// Declare the ring buffer as a static — this is how maps work in eBPF
#[map]
static EVENTS: RingBuf = RingBuf::with_byte_size(8 * 4096, 0);
// Attach to sched:sched_switch
#[tracepoint]
pub fn sched_switch(ctx: TracePointContext) -> u32 {
// The tracepoint payload layout for sched_switch on Linux 5.x/6.x:
// (after the common tracepoint header)
// offset 0: prev_comm char[16] (TASK_COMM_LEN)
// offset 16: prev_pid u32
// offset 20: prev_prio u32
// offset 24: prev_state u64 (TASK_* state mask)
// offset 32: next_comm char[16]
// offset 48: next_pid u32
// offset 52: next_prio u32
//
// Verify on your system: cat /sys/kernel/tracing/events/sched/sched_switch/format
let prev_pid = unsafe { ctx.read_at::<u32>(16).unwrap_or(0) };
let next_pid = unsafe { ctx.read_at::<u32>(48).unwrap_or(0) };
let cpu_id = unsafe { bpf_get_smp_processor_id() };
let timestamp = unsafe { bpf_ktime_get_ns() };
let event = SchedulerEvent {
cpu_id,
prev_pid,
next_pid,
timestamp,
};
// Send to ring buffer — userspace reads from the EVENTS map
EVENTS.output(&event, 0);
0
}
}A few things to notice here:
#![no_std] and #![no_main]: eBPF programs don’t use the standard library (no heap, no I/O) and don’t have a main function. The entry point is the function marked with #[tracepoint].
#[map]: The #[map] attribute registers the static as an eBPF map. The ring buffer is declared at the top of the file and lives for the lifetime of the program. You don’t access it through a context object.
#[tracepoint]: The eBPF macro marks the function as a tracepoint program. No arguments — the category and name are provided from userspace via program.attach(). The eBPF macros are lowercase; userspace program types are PascalCase.
ctx: TracePointContext: The context is passed by value (not &mut). The TracePointContext gives you access to the tracepoint payload via read_at::<T>(offset).
unsafe { ctx.read_at::<T>(offset) }: Reading tracepoint payload requires unsafe — the verifier can’t guarantee the memory is valid. In practice, reading from a kernel-placed tracepoint payload is safe.
bpf_get_smp_processor_id() and bpf_ktime_get_ns(): These are BPF helpers exposed through aya_ebpf::helpers. They’re available in every eBPF program.
Verifying tracepoint offsets on your kernel. The
sched_switchlayout above is correct for Linux 5.x and 6.x, but kernel versions can change field sizes, add new fields, or reorder them. Before you trust any hardcoded offset, check the format file for your running kernel:cat /sys/kernel/tracing/events/sched/sched_switch/formatYou’ll see output like this (your exact offsets may differ):
name: sched_switch ID: 314 format: field:unsigned short common_type; offset:0; size:2; signed:0; field:unsigned char common_flags; offset:2; size:1; signed:0; field:unsigned char common_preempt_count; offset:3; size:1; signed:0; field:int common_pad; offset:4; size:4; signed:1; field:char prev_comm[16]; offset:8; size:16; signed:0; field:pid_t prev_pid; offset:24; size:4; signed:1; field:int prev_prio; offset:28; size:4; signed:1; field:long prev_state; offset:32; size:8; signed:1; field:char next_comm[16]; offset:40; size:16; signed:0; field:pid_t next_pid; offset:56; size:4; signed:1; field:int next_prio; offset:60; size:4; signed:1;The first four fields (
common_*) are the tracepoint header — 8 bytes of metadata present in every tracepoint record. Theread_atoffsets in this tutorial start from the first byte after this header, so they are 8 bytes less than the offsets shown in the format file. For example, the format file showsprev_pidat offset 24, and the correspondingread_atcall uses offset 16 (24 − 8 = 16).The key cross-check: find the field you want in the format output, read its
offsetvalue, and subtract 8 to get yourread_atoffset. If the result doesn’t match the tutorial’s code, the field layout has changed on your kernel — and you’ll read garbage unless you update the offset. This applies to every tracepoint in every part of this tutorial. When in doubt, check the format file.
User-Space Program (perf-monitor/)
The user-space program loads the compiled eBPF object, creates maps, attaches programs, and reads data from ring buffers. It runs as a normal Rust binary:
// perf-monitor/src/main.rs
use aya::programs::TracePoint;
use aya::maps::RingBuf;
use aya::Ebpf;
use std::convert::TryFrom;
#[derive(Clone, Copy, Debug)]
#[repr(C)]
struct SchedulerEvent {
cpu_id: u32,
prev_pid: u32,
next_pid: u32,
timestamp: u64,
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// The eBPF object is embedded at compile-time via OUT_DIR.
// Ebpf::load() finds it without needing a file path.
let mut ebpf = aya::Ebpf::load(aya::include_bytes_aligned!(
concat!(env!("OUT_DIR"), "/perf-monitor")
))?;
// Attach the tracepoint program
let program: &mut TracePoint = ebpf
.program_mut("perf_monitor")
.unwrap()
.try_into()?;
program.load()?;
program.attach("sched", "sched_switch")?;
// Create the ring buffer from the map named "events"
let mut ring_buf = RingBuf::try_from(ebpf.map_mut("events")?)?;
// Poll the ring buffer
loop {
tokio::time::sleep(tokio::time::Duration::from_millis(100)).await;
while let Some(item) = ring_buf.next() {
// item derefs to &[u8] — cast to our event type
let event = unsafe {
std::ptr::read_unaligned(item.as_ptr() as *const SchedulerEvent)
};
println!(
"cpu={} prev_pid={} next_pid={} ts={}",
event.cpu_id, event.prev_pid, event.next_pid, event.timestamp
);
}
}
}Note: the program name in the template is perf_monitor (underscore, from the crate name). The map name events comes from the eBPF side — check perf-monitor-ebpf/src/main.rs for the map definition.
Why is SchedulerEvent defined twice? The eBPF program and the user-space program each define their own copy of the event struct. This looks wrong — the perf-monitor-common crate exists for shared types. In a real project, you’d put SchedulerEvent in the common crate and import it from both sides. The examples here define it inline for two reasons: (1) the eBPF crate uses #![no_std], so the common type can’t derive Debug or use any std types — you’d need separate Debug-less and Debug-ful versions or conditional compilation, and (2) keeping each code block self-contained means you can copy any block into a file and it compiles. For this tutorial’s scope, the inline approach keeps things simple. If you’re building a production monitor, move shared types into the common crate and use conditional compilation for Debug.
Running Locally
You’ll need a Linux machine with eBPF support (kernel 5.8+). If you’re developing on a VM, eBPF may or may not work depending on the hypervisor — nested virtualization support for eBPF varies. On real hardware it always works.
# Build both userspace and eBPF — build.rs compiles eBPF and embeds it
cargo build
# Run as root (eBPF programs require elevated privileges)
sudo ./target/debug/perf-monitor
The eBPF programs are compiled automatically by the build.rs script in the perf-monitor-ebpf/ crate. cargo build handles everything in one step.
Troubleshooting
Build fails with exit status around 25856 → The nightly toolchain can’t compile for bpfel-unknown-none. Run:
rustup target add bpfel-unknown-none --toolchain nightly
Build fails with “target not found” for bpfel-unknown-none → Same fix. The BPF target isn’t automatically installed when you add the nightly toolchain — you have to add it explicitly.
Permission denied when running → eBPF programs require root. Use sudo.
Project Structure for This Tutorial
For the full performance monitor, we’ll extend the scaffold with multiple eBPF programs and multiple data sources. The structure we’ll build:
perf-monitor/ ← workspace root
├── perf-monitor/ ← user-space program
│ └── src/
│ ├── main.rs ← event loop, loads programs, aggregates data
│ ├── pmc.rs ← perf_event_open wrapper, PMC event reading
│ ├── numa.rs ← procfs/sysfs readers for NUMA stats
│ ├── thermal.rs ← sysfs thermal zone reader
│ └── types.rs ← shared event structs
├── perf-monitor-ebpf/ ← eBPF programs
│ └── src/
│ ├── scheduler.rs ← sched tracepoints: switch, waking, stat_wait
│ ├── blockio.rs ← block I/O tracepoints
│ ├── vhost.rs ← kprobes on vhost/virtio ring functions
│ └── lib.rs ← map definitions, program registration
└── perf-monitor-common/ ← code shared between userspace and eBPF
└── src/lib.rs
Next: Part 3 — Hardware PMCs with perf_event_open — Open a counter, read it, and compute instructions per cycle.
Part 3 — Hardware PMCs with perf_event_open
The CPU has hardware counters on-die. The only way to read them on Linux is perf_event_open.
What PMCs Are
Performance Monitoring Counters — PMCs — are tiny registers inside the CPU chip. They count specific microarchitectural events: a cache line loaded from L1, a branch instruction resolved, a TLB walk performed. Every modern x86 and ARM processor has them.
The counter is an accumulation of incremental events. Each hardware event increments it. You open a file descriptor for a specific event, and then you read the counter value by reading from that file descriptor. That’s the whole interface.
The events are defined by a type and a config. On x86, type is usually PERF_TYPE_HARDWARE (0) or PERF_TYPE_RAW (4). The hardware events — instructions retired, CPU cycles, cache references — live in the hardware type. The raw events — cache misses, branch mispredicts, TLB walks — live in the raw type, and their event numbers vary by CPU microarchitecture. That’s why Part 4 matters: before you can open a cache miss counter, you need to know what CPU you’re on.
The syscall
perf_event_open is a Linux-specific syscall. Here’s its signature from the kernel headers:
int perf_event_open(
struct perf_event_attr *attr, // what to count
pid_t pid, // attach to this process (0 = self)
int cpu, // which CPU (-1 = all)
int group_fd, // group leader fd (-1 = new group)
unsigned long flags // PERF_FLAG_FD_CLOEXEC etc.
);
Returns a file descriptor on success, or -1 on error.
The perf_event_attr struct is the interesting part. Here’s the relevant subset from the kernel headers (include/uapi/linux/perf_event.h):
struct perf_event_attr {
__u32 type; // PERF_TYPE_HARDWARE, PERF_TYPE_RAW, etc.
__u32 size; // sizeof(struct perf_event_attr)
__u64 config; // which specific event
union {
__u64 sample_period; // sample every N events (for sampling mode)
__u64 sample_freq; // target sample frequency (for sampling mode)
};
__u64 sample_type; // what gets written to the sample buffer
__u64 read_format; // format for reading counter values
__u64 flags; // disabled, pinned, inherit, etc.
};
For counting mode (what we’ll use), you set the union to 0 — both sample_period and sample_freq are zero, meaning no sampling. The kernel won’t generate PMIs (Performance Monitoring Interrupts), and you read the counter value directly from the file descriptor. For profiling mode, you set sample_freq to a target sample rate (e.g., 1000 Hz) and the kernel samples periodically.
The Rust hand-rolled struct below uses a single sample_period field instead of the union — since we’re in counting mode, the union value is always 0 and a single field is sufficient. The struct includes size (required by the kernel) and omits read_format and the full flags field in the explanation above — the working code at the end of this part includes everything you need.
Rust bindings
There’s a perf-event crate that wraps perf_event_open with a safe interface, but it doesn’t support all the raw PMC event types we need, and libc::perf_event_attr isn’t available on all platforms. So we hand-roll the struct and call the syscall directly via libc.
On Linux, perf_event_open is syscall number 298 on x86_64. The number varies by architecture: 241 on ARM64 and RISC-V, 319 on PowerPC. The code uses libc::SYS_perf_event_open, which resolves to the right number per architecture. You can’t call it through libc::perf_event_open because not all libc implementations expose it.
The full open_pmc function is in the minimal working example at the end of this part. The next few sections show the API surface — what the events are, how to read counters, how to compute useful metrics — and then we bring it all together.
Opening a counter for instructions and cycles
The hardware events live in PERF_TYPE_HARDWARE. The two universal hardware events — every CPU supports them — are PERF_COUNT_HW_INSTRUCTIONS and PERF_COUNT_HW_CPU_CYCLES:
#![allow(unused)]
fn main() {
let instr_fd = open_pmc(
libc::PERF_TYPE_HARDWARE as u32,
libc::PERF_COUNT_HW_INSTRUCTIONS as u64,
0, // attach to current process
-1, // all CPUs
)?;
let cycles_fd = open_pmc(
libc::PERF_TYPE_HARDWARE as u32,
libc::PERF_COUNT_HW_CPU_CYCLES as u64,
0,
-1,
)?;
}Reading and resetting
Reading from the file descriptor returns the current counter value. The counter starts disabled — you enable it with an ioctl call:
#![allow(unused)]
fn main() {
fn read_counter(fd: i32) -> std::io::Result<u64> {
let mut val: u64 = 0;
let n = unsafe {
libc::read(fd, &mut val as *mut _ as *mut libc::c_void, 8)
};
if n < 0 {
Err(std::io::Error::last_os_error())
} else {
Ok(val)
}
}
fn enable_counter(fd: i32) -> std::io::Result<()> {
let ret = unsafe {
// arg=0 enables this counter. PERF_IOC_FLAG_GROUP would enable
// all counters in the same group — but we aren't using groups here.
libc::ioctl(fd, libc::PERF_EVENT_IOC_ENABLE, 0)
};
if ret < 0 {
return Err(std::io::Error::last_os_error());
}
Ok(())
}
}To reset the counter to zero, use ioctl with PERF_EVENT_IOC_RESET:
#![allow(unused)]
fn main() {
unsafe {
libc::ioctl(fd, libc::PERF_EVENT_IOC_RESET, 0);
}
}What happens without pinned (counter multiplexing)
The CPU has a limited number of PMC slots — typically 2–8 per core, depending on the microarchitecture. If you open more counters than the hardware has slots, something has to give.
With pinned=1, the kernel refuses to schedule the counter if no slot is available. The open_pmc call succeeds, but when you enable the counter, the kernel sets time_running = 0 and read_counter returns 0. You get an error on ioctl(PERF_EVENT_IOC_ENABLE) instead — the counter was never scheduled. This is the fail-fast behavior: you know immediately that the counter couldn’t run.
With pinned=0, the kernel multiplexes — it time-shares the PMC slots among all the counters that want them. Each counter gets scheduled for a slice of time, then rotated out. The raw counter value you read is only the count accumulated during the slices the counter was actually on the PMU, not the full interval.
To get the correct value, you must scale:
scaled_value = raw_value * time_enabled / time_running
time_enabled is how long the counter was supposed to be running. time_running is how long it actually was on the PMU. When there’s no multiplexing, they’re equal and the scaling is 1:1 — the raw value is already correct.
To read these fields, set read_format to PERF_FORMAT_TIME_ENABLED | PERF_FORMAT_TIME_RUNNING in the perf_event_attr struct:
#![allow(unused)]
fn main() {
const PERF_FORMAT_TOTAL_TIME_ENABLED: u64 = 1 << 0; // bit 0
const PERF_FORMAT_TOTAL_TIME_RUNNING: u64 = 1 << 1; // bit 1
// read_format = TOTAL_TIME_ENABLED | TOTAL_TIME_RUNNING
let read_format = PERF_FORMAT_TOTAL_TIME_ENABLED | PERF_FORMAT_TOTAL_TIME_RUNNING;
}These names and bit positions come directly from the kernel header (include/uapi/linux/perf_event.h). The TOTAL_ prefix distinguishes them from the per-sample equivalents used in sampling mode.
Then read(fd, ...) returns a struct instead of a single u64:
#![allow(unused)]
fn main() {
#[repr(C)]
struct ReadFormatScaled {
value: u64, // the raw counter value
time_enabled: u64, // how long the counter was enabled (nanoseconds)
time_running: u64, // how long the counter was actually on the PMU (nanoseconds)
}
fn read_counter_scaled(fd: i32) -> std::io::Result<(u64, f64)> {
let mut buf = ReadFormatScaled { value: 0, time_enabled: 0, time_running: 0 };
let n = unsafe {
libc::read(
fd,
&mut buf as *mut _ as *mut libc::c_void,
std::mem::size_of::<ReadFormatScaled>(),
)
};
if n < 0 {
return Err(std::io::Error::last_os_error());
}
let scaled = if buf.time_running > 0 {
buf.value as f64 * buf.time_enabled as f64 / buf.time_running as f64
} else {
0.0 // counter was never scheduled
};
Ok((buf.value, scaled))
}
}The scaling factor time_enabled / time_running corrects for the missing counts. If a counter was scheduled for 50% of the interval (time_running = time_enabled / 2), the raw count is approximately half the true count — scaling doubles it.
This tutorial uses pinned=1 everywhere. Pinned counters give exact values, no scaling needed. But if you’re building a tool that opens many counters simultaneously (more than the PMU has slots), you’ll hit multiplexing. Use the scaled read when pinned=0 — otherwise your counts will be silently low.
Computing IPC and stall ratio
Instructions per cycle (IPC) tells you how much useful work the CPU is doing per clock tick. A healthy compute-bound workload might hit 3-4 IPC on a modern out-of-order core. A memory-bound workload stalls frequently and might hit 0.5 IPC.
#![allow(unused)]
fn main() {
fn compute_ipc(instructions: u64, cycles: u64) -> f64 {
if cycles == 0 {
return 0.0;
}
instructions as f64 / cycles as f64
}
}Stall ratio is the fraction of cycles where the core wasn’t retiring instructions. This happens when the core is waiting on memory, a branch mispredict, or any other pipeline stall.
The key insight: when the pipeline stalls, the instruction counter slows down even though cycles keep ticking. Here’s the catch — modern out-of-order cores can retire multiple instructions per cycle (4-6 on recent x86). So cycles - instructions doesn’t directly give you stalled cycles. When IPC < 1, the gap cycles - instructions is a lower bound on stalls — cycles where the core couldn’t even manage 1 retirement. When IPC > 1, the formula returns 0 — it can’t detect stalls at all. A core that retires 4 instructions in half its cycles and 0 in the other half (stall bursts) still shows IPC = 2, which looks healthy even though half the cycles were stalls. The average hides the bursts.
#![allow(unused)]
fn main() {
fn compute_stall_ratio(instructions: u64, cycles: u64) -> f64 {
if instructions == 0 || cycles == 0 {
return 0.0;
}
// When IPC < 1, the gap is a lower bound on stalled cycles.
// When IPC > 1, this returns 0 — stalls are hidden by the average.
// See the "Computing IPC and stall ratio" section for the full story.
let stalled = (cycles as i64 - instructions as i64).max(0) as f64;
stalled / cycles as f64
}
}This is a simplification. It can’t detect stall bursts hidden by a high average IPC, and some cycles are legitimately empty (no instructions ready, no work to do). For a more accurate stall ratio, you’d read the actual stall-cycle PMCs that modern CPUs expose (e.g., INT_MISC.RECOVERY_CYCLES on Intel). But as a quick health check from two counters alone, it’s a useful signal.
Per-process vs. system-wide monitoring
The pid parameter controls what gets counted. The cpu parameter controls where. Getting these wrong gives you the right numbers for the wrong thing.
Per-process (pid=0, cpu=-1): Open the counter with pid=0 and you measure the calling process as it runs on any CPU. The kernel follows the process across CPU migrations and accumulates the count.
#![allow(unused)]
fn main() {
let cycles_fd = open_pmc(PERF_TYPE_HARDWARE, libc::PERF_COUNT_HW_CPU_CYCLES as u64, 0, -1)?;
}This is what the example program uses. It works for measuring the IPC of a workload you launch from your monitoring tool. It does not measure the entire system — if your process is mostly sleeping (waiting for the next read interval), the counter values will be near zero.
System-wide (pid=-1, per-CPU): Open a counter for each CPU with pid=-1 to measure all processes on that CPU. The kernel requires a specific cpu number when pid=-1 (passing cpu=-1 with pid=-1 returns EINVAL):
#![allow(unused)]
fn main() {
fn open_all_cpus(type_: u32, config: u64) -> std::io::Result<Vec<(i32, i32)>> {
let mut fds = Vec::new();
for cpu in 0..num_cpus() { // num_cpus() from the `num_cpus` crate, or read /sys/devices/system/cpu/online
// pid=-1: system-wide. Measures all processes on this CPU.
let fd = open_pmc(type_, config, -1, cpu as i32)?;
fds.push((cpu, fd));
}
Ok(fds)
}
}Read each fd and aggregate in userspace for a total. Or keep them separate for per-CPU granularity — a CPU with unusually high cache misses might be running a memory-bound workload pinned to that core.
For a system monitoring tool, pid=-1 is almost always what you want. The example program uses pid=0 for simplicity — a single fd, no aggregation loop — but a real deployment should switch to pid=-1 with per-CPU fds. Part 8 uses pid=-1 for uncore IMC counters, which are inherently system-wide.
Error handling
Two errors are common:
EPERM: The syscall returns EPERM when the calling process lacks the right capabilities. PMC access requires CAP_SYS_ADMIN (root) or CAP_SYS_PERFMON (Linux 5.8+, a narrower capability). If you hit this in a container, the host may need to grant the capability.
EINVAL: The event you asked for isn’t available on this CPU. Raw PMC events in particular vary by microarchitecture — the same event number might mean “cache miss” on Skylake and “not defined” on Ice Lake. This is why Part 4 exists: detect the CPU first, then select events.
A minimal working example
Here’s a complete program that opens an instructions counter and a cycles counter, enables them, reads them once per second, and prints per-second IPC. This brings together the open_pmc, read_counter, and enable_counter functions from the sections above:
use std::io;
use std::thread;
use std::time::Duration;
fn open_pmc(type_: u32, config: u64, pid: i32, cpu: i32) -> io::Result<i32> {
// Hand-rolled perf_event_attr — libc::perf_event_attr is not available
// on all platforms. The field ordering must match the kernel struct exactly.
#[repr(C)]
struct PerfEventAttr {
type_: u32,
size: u32,
config: u64,
sample_period: u64, // 0 for counting mode
sample_type: u64,
read_format: u64,
flags: u64, // bit 0: disabled, bit 2: pinned
}
let attr = PerfEventAttr {
type_,
size: std::mem::size_of::<PerfEventAttr>() as u32,
config,
sample_period: 0, // counting mode: no sampling, read counter directly
sample_type: 0,
read_format: 0,
flags: 0b101, // disabled=1 (bit 0), pinned=1 (bit 2)
//
// The flags field is a C bitfield packed into a u64. On x86-64,
// GCC/Clang allocate bits from LSB to MSB, so:
// bit 0 = disabled (start counter in disabled state)
// bit 1 = inherit (children inherit the counter — not set here)
// bit 2 = pinned (counter must stay on the PMU — prevents multiplexing)
// We set disabled so we can enable the counter explicitly via ioctl.
// We set pinned because counting mode needs the counter scheduled
// at all times — without pinned, the kernel may multiplex the counter
// on busy systems, producing scaled values instead of exact counts.
};
let fd = unsafe {
libc::syscall(
libc::SYS_perf_event_open,
&attr as *const _,
pid,
cpu,
-1, // no group leader
0, // no flags
)
};
if fd < 0 {
return Err(io::Error::last_os_error());
}
Ok(fd as i32)
}
fn read_counter(fd: i32) -> io::Result<u64> {
let mut val: u64 = 0;
let n = unsafe {
libc::read(fd, &mut val as *mut _ as *mut libc::c_void, 8)
};
if n < 0 {
Err(io::Error::last_os_error())
} else {
Ok(val)
}
}
fn enable_counter(fd: i32) -> io::Result<()> {
let ret = unsafe {
// arg=0: enable this counter (not a group)
libc::ioctl(fd, libc::PERF_EVENT_IOC_ENABLE, 0)
};
if ret < 0 {
return Err(io::Error::last_os_error());
}
Ok(())
}
fn main() -> io::Result<()> {
let instr_fd = open_pmc(
libc::PERF_TYPE_HARDWARE as u32,
libc::PERF_COUNT_HW_INSTRUCTIONS as u64,
0, // pid=0: measure this process (not system-wide; see "Per-process vs. system-wide" below)
-1, // follow this process across all CPUs
)?;
let cycles_fd = open_pmc(
libc::PERF_TYPE_HARDWARE as u32,
libc::PERF_COUNT_HW_CPU_CYCLES as u64,
0, // pid=0: measure this process
-1, // follow this process across all CPUs
)?;
enable_counter(instr_fd)?;
enable_counter(cycles_fd)?;
let mut prev_instr = 0u64;
let mut prev_cycles = 0u64;
loop {
thread::sleep(Duration::from_secs(1));
let instr = read_counter(instr_fd)?;
let cycles = read_counter(cycles_fd)?;
let instr_delta = instr - prev_instr;
let cycles_delta = cycles - prev_cycles;
let ipc = if cycles_delta > 0 {
instr_delta as f64 / cycles_delta as f64
} else {
0.0
};
println!("instructions={instr_delta} cycles={cycles_delta} ipc={ipc:.3}");
prev_instr = instr;
prev_cycles = cycles;
}
}Run it with sudo cargo run — it needs root. The counter values are cumulative, so the program computes deltas between reads to get per-second IPC.
Next: Part 4 — CPU Microarchitecture Detection — Before you open any raw PMC, figure out what CPU you’re running on and pick the right event numbers.
Part 4 — CPU Microarchitecture Detection
The same PMC event number means different things on different CPUs. On Intel Skylake, event 0xD1 with umask 0x08 counts L1 data cache load misses. On AMD Zen 2, that same event number might not even exist. On ARM, the encoding scheme is completely different. If you hardcode event numbers, your code breaks on every machine that isn’t yours.
The fix isn’t complicated: before you open any PMC, figure out what CPU you’re running on, then pick the right event numbers for that chip. This part builds the detection layer that Parts 5 through 12 will rely on.
Reading /proc/cpuinfo
The file has one block per CPU. For our purposes, the relevant fields are the same across all cores of the same physical CPU, so we’ll read the first block:
#![allow(unused)]
fn main() {
use std::fs;
fn read_cpuinfo() -> io::Result<String> {
fs::read_to_string("/proc/cpuinfo")
}
}Here’s what the output looks like on an Intel Skylake desktop:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 94
model name : Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
stepping : 3
microcode : 0x96
cpu MHz : 4000.000
cache size : 8192 KB
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
The fields we care about:
vendor_id:"GenuineIntel"or"AuthenticAMD"(the strings Linux uses)cpu family: a decimal number, part of the CPUID family fieldmodel: decimal, the CPUID model fieldstepping: decimal, the CPUID stepping fieldflags: a space-separated list of CPU feature flags
The family and model numbers are how we identify the microarchitecture. For Intel, family 6 means “P6 family or later” (which covers everything from Pentium Pro through modern Skylake/Ice Lake). The model field then distinguishes the specific generation.
Parsing vendor and family/model
#![allow(unused)]
fn main() {
use std::io::{self, BufRead};
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Vendor {
Intel,
Amd,
Unknown,
}
#[derive(Debug, Clone)]
pub struct CpuInfo {
pub vendor: Vendor,
pub family: u32,
pub model: u32,
pub stepping: u32,
pub flags: Vec<String>,
}
fn parse_cpuinfo(raw: &str) -> Option<CpuInfo> {
let mut vendor = Vendor::Unknown;
let mut family: u32 = 0;
let mut model: u32 = 0;
let mut stepping: u32 = 0;
let mut flags: Vec<String> = Vec::new();
for line in raw.lines() {
let mut parts = line.splitn(2, ':');
let key = parts.next()?.trim();
let value = parts.next()?.trim();
match key {
"vendor_id" | "vendor" => {
vendor = match value {
"GenuineIntel" => Vendor::Intel,
"AuthenticAMD" => Vendor::Amd,
_ => Vendor::Unknown,
};
}
"cpu family" => {
family = value.parse().ok()?;
}
"model" => {
model = value.parse().ok()?;
}
"stepping" => {
stepping = value.parse().ok()?;
}
"flags" => {
flags = value.split_whitespace().map(String::from).collect();
}
_ => {}
}
// Stop after the first processor block. "processor" is the first
// field of each CPU block in /proc/cpuinfo. When we see the second
// "processor" line and we've already collected flags from the first
// block, we've read everything we need (vendor, family, model,
// stepping, and flags are the same across all cores of the same CPU).
if key == "processor" && !flags.is_empty() {
break;
}
}
if vendor == Vendor::Unknown {
return None;
}
Some(CpuInfo { vendor, family, model, stepping, flags })
}
}Mapping to microarchitecture names
The family and model numbers combine to produce a “microarchitecture identifier.” Here’s how to build that mapping for Intel:
#![allow(unused)]
fn main() {
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Microarch {
// Intel
Skylake,
SkylakeSp,
KabyLake, // includes Coffee Lake, Comet Lake (same PMC event encodings)
IceLake,
IceLakeSp,
RocketLake,
AlderLake,
SapphireRapids,
// AMD
Zen,
ZenPlus,
Zen2,
Zen3,
Zen4,
Zen5,
// Generic
Unknown,
}
pub fn detect_intel_microarch(family: u32, model: u32, stepping: u32) -> Microarch {
// The "model" field in /proc/cpuinfo is the CPUID model value with the
// extended model already folded in (family 6 Intel: model = ext_model << 4 | base_model).
//
// Why do some very different CPUs share a match arm? Because they share PMC events.
// Kaby Lake, Coffee Lake, and Comet Lake all use Skylake-core PMCs. The perfmon
// repo (github.com/intel/perfmon) puts them all in the /SKL/ event directory.
// For a monitoring tool, the PMC mapping is what matters, not the marketing name.
match (family, model) {
// Skylake family — all use the same PMC event encodings
(6, 85) => Microarch::SkylakeSp, // Skylake SP / Cascade Lake SP
(6, 94) => Microarch::Skylake, // Desktop Skylake (i7-6700K)
(6, 142 | 158) => Microarch::KabyLake, // Kaby Lake / Coffee Lake (same PMCs)
(6, 165 | 166) => Microarch::KabyLake, // Comet Lake H/S (same PMCs as KBL)
// Ice Lake — some events moved, L3 miss not available on all SKUs
(6, 125 | 126) => Microarch::IceLake, // Ice Lake client (0x7D, 0x7E)
(6, 106 | 108) => Microarch::IceLakeSp, // Ice Lake server (0x6A, 0x6C)
// Tiger Lake — Ice Lake PMC events, plus some uncore changes
(6, 140 | 141) => Microarch::IceLake, // Tiger Lake (0x8C, 0x8D; ICL PMCs)
// Rocket Lake
(6, 167) => Microarch::RocketLake, // Rocket Lake (0xA7)
// Alder Lake — hybrid P-core (Golden Cove) + E-core (Gracemont)
(6, 151 | 154) => Microarch::AlderLake, // Alder Lake desktop (0x97) / mobile (0x9A)
// Sapphire Rapids
(6, 143) => Microarch::SapphireRapids, // Sapphire Rapids (0x8F)
_ => Microarch::Unknown,
}
}
pub fn detect_amd_microarch(family: u32, model: u32) -> Microarch {
// AMD family encoding: CPUID family = extended_family * 16 + base_family
// For Zen, base_family is 15 (0xF) with extended_family = 1 → CPUID family 23 (0x17)
// For Zen 2, AMD kept family 23 (0x17) for Matisse/Rome desktop and server parts.
// Zen 3 and Zen 4 use family 25 (0x19).
// Zen 5 uses family 26 (0x1A).
//
// Model numbers come from the AMD Processor Programming Reference (PPR) and
// the /proc/cpuinfo "model" field. Server and desktop parts within the same
// Zen generation use different model numbers but share PMC event encodings.
match (family, model) {
// Zen 1
(23, 1) => Microarch::Zen, // Naples (EPYC 1st gen)
(23, 17) => Microarch::Zen, // Raven Ridge (Zen 1 APU)
// Zen+
(23, 8) => Microarch::ZenPlus, // Pinnacle Ridge (Ryzen 2000 desktop)
(23, 32) => Microarch::ZenPlus, // Colfax (Threadripper 2000)
// Zen 2
(23, 49) => Microarch::Zen2, // Rome (EPYC 2nd gen, model 0x31)
(23, 113) => Microarch::Zen2, // Matisse (Ryzen 3000 desktop, model 0x71)
// Zen 3
(25, 1) => Microarch::Zen3, // Milan (EPYC 3rd gen)
(25, 33) => Microarch::Zen3, // Vermeer (Ryzen 5000 desktop, model 0x21)
// Zen 4
(25, 17) => Microarch::Zen4, // Genoa (EPYC 4th gen, model 0x11)
(25, 97) => Microarch::Zen4, // Raphael (Ryzen 7000 desktop, model 0x61)
// Zen 5 — family 26 (0x1A)
// Launched July 2024 (mobile), August 2024 (desktop), October 2024 (server).
// Model numbers from AMD PPR and the Linux kernel's cpu_has_entrysign()
// (arch/x86/kernel/cpu/microcode/amd.c), which lists family 0x1A models in
// three ranges: ≤0x2f, 0x40–0x4f, 0x60–0x7f.
(26, 1) => Microarch::Zen5, // Turin (EPYC 9005 server, model 0x01)
(26, 36) => Microarch::Zen5, // Strix Point (Ryzen AI 300 mobile, model 0x24)
(26, 68) => Microarch::Zen5, // Granite Ridge (Ryzen 9000 desktop, model 0x44)
_ => Microarch::Unknown,
}
}
pub fn detect_microarch(info: &CpuInfo) -> Microarch {
match info.vendor {
Vendor::Intel => detect_intel_microarch(info.family, info.model, info.stepping),
Vendor::Amd => detect_amd_microarch(info.family, info.model),
Vendor::Unknown => Microarch::Unknown,
}
}
}This mapping covers the major Intel and AMD microarchitectures in production. Two caveats for production use:
-
Intel hybrid cores (Alder Lake and later): P-cores use Golden Cove PMC encodings; E-cores use Gracemont encodings. The
detect_intel_microarchfunction returnsAlderLakefor both, but a production tool would need to detect which core type it’s running on (via thehybridCPUID leaf, leaf 0x1A) and select events per-core. The Intel SDM documents both encoding sets in Volume 3B, Chapter 19. -
Extended model numbers: The
modelfield in/proc/cpuinfoalready folds in the extended model bits for family 6 Intel CPUs (the kernel does this for you). If you’re reading CPUID directly, you need to combineext_model << 4 | base_modelyourself.
For production code, cross-reference against the Intel SDM (Software Developer’s Manual, Volume 3B, Chapter 19) or use a maintained lookup table from a project like the Intel perfmon repository (github.com/intel/perfmon). But for a monitoring tool, the key is having some mapping, not a complete one.
Why this matters for PMC events
Here’s a concrete example. On Intel Skylake, L1 data cache load miss is:
- Type:
PERF_TYPE_RAW(4) - Event:
0xD1 - Umask:
0x08
On Ice Lake, the same counter exists but some events moved. The safe approach is to enumerate available events on the target machine rather than hardcoding.
Enumerating available events with perf list
Before opening any raw event, you can ask the kernel what events are available:
perf list
This prints a categorized list of events. For raw events, look for entries under raw in the output. To narrow it down:
perf list | grep -i cache | head -20
To get only raw events with their hex encodings:
perf list | grep -i "raw" | head -20
Each raw event has a hex encoding in the output — that’s the config value to use with PERF_TYPE_RAW.
From a Rust program, you can call perf list via std::process::Command and parse the output. Or use the sysfs path directly:
#![allow(unused)]
fn main() {
use std::fs;
fn list_raw_events() -> io::Result<Vec<String>> {
let path = "/sys/bus/event_source/devices/cpu/events";
let mut events = Vec::new();
if let Ok(entries) = fs::read_dir(path) {
for entry in entries.flatten() {
if let Ok(content) = fs::read_to_string(entry.path()) {
// Each file is a perf event definition in the format:
// event=0xD1\numask=0x08\n
events.push(entry.file_name().into_string().unwrap_or_default());
}
}
}
Ok(events)
}
}Putting it together
A helper that returns everything we need:
#![allow(unused)]
fn main() {
pub struct CpuMicroarch {
pub cpuinfo: CpuInfo,
pub microarch: Microarch,
pub vendor: Vendor,
}
pub fn detect() -> io::Result<CpuMicroarch> {
let raw = fs::read_to_string("/proc/cpuinfo")?;
let cpuinfo = parse_cpuinfo(&raw).ok_or_else(|| {
io::Error::new(io::ErrorKind::InvalidData, "could not parse cpuinfo")
})?;
let microarch = detect_microarch(&cpuinfo);
let vendor = cpuinfo.vendor.clone();
Ok(CpuMicroarch { cpuinfo, microarch, vendor })
}
}Next: Part 5 — Cache and TLB Metrics from PMC — Open cache miss and TLB walk counters, compute miss rates, and select the right events for your CPU.
Part 5 — Cache and TLB Metrics from PMC
L1 cache misses cost a few cycles. L3 misses cost a few hundred.
That’s not hyperbole — the difference between a cache hit and a cache miss at each level of the hierarchy is an order of magnitude larger. A L1 miss that hits in L2 might cost 10 cycles. A L3 miss that goes to main memory costs 100-300 cycles depending on the system. Once you see those numbers in your metrics, memory-bound workloads become obvious.
The cache hierarchy in plain language
Modern CPUs have several levels of cache. Each core has its own private L1 — a small, fast cache split into L1 data and L1 instructions. L2 is also private to each core but larger and slower. L3 (or LLC — Last Level Cache) is typically shared across cores on the same chip and slower still.
When the core needs a piece of data, it checks L1 first. If it finds the cache line, that’s a hit and the data is available in 1-2 cycles. If L1 misses, it checks L2. If L2 misses, it checks L3. If L3 misses, it goes to main memory — a round-trip that might be 100-300 nanoseconds on a fast system.
Each level is a separate performance counter in the CPU. Counting L3 misses tells you how often the CPU is going to main memory.
PMC events for each cache level
On Intel x86, raw events are identified by a type and a pair of hex numbers: the event selector and the unit mask (umask). The format for raw events in perf:
event=0x<EventHex>,umask=0x<UmaskHex>
When you use perf_event_open with PERF_TYPE_RAW (4), you set config to (umask << 8) | event.
Here are the Intel Skylake cache events. These come from the MEM_LOAD_RETIRED event family (event 0xD1) and the TLB event families (0x08 for dTLB, 0x85 for iTLB). The event numbers are from the Intel SDM (Software Developer’s Manual, Volume 3B, Chapter 19). Verify these on your target system with perf list — some events are SKU-dependent.
| Metric | Event | Umask | Intel SDM Name |
|---|---|---|---|
| L1 dcache load miss | 0xD1 | 0x08 | MEM_LOAD_RETIRED.L1_MISS |
| L2 cache miss | 0xD1 | 0x10 | MEM_LOAD_RETIRED.L2_MISS |
| L3 cache miss | 0xD1 | 0x20 | MEM_LOAD_RETIRED.L3_MISS |
| L1 dcache load hit | 0xD1 | 0x01 | MEM_LOAD_RETIRED.L1_HIT |
The L3 miss counter requires SKU verification — some Intel parts don’t expose it. Check perf list on your target system before relying on it.
A note on event accuracy. MEM_LOAD_RETIRED counts retired load instructions — loads that completed. This means it doesn’t count speculative loads that were issued but discarded (e.g., on a mispredicted branch). For most monitoring use cases, retired loads are what you want: they reflect the work the program actually did, not work it speculated about and threw away. If you need to count all load accesses including speculative ones, use the MEM_INST_RETIRED event family instead — but the distinction usually only matters for profiling, not metrics.
The TLB structure
The Translation Lookaside Buffer (TLB) is a hardware cache for virtual-to-physical address translations. Every memory access needs an address translation: virtual address → physical address. The TLB caches these translations so the CPU doesn’t have to walk the page table every time.
There are two TLBs:
- dTLB (data TLB): caches translations for memory reads and writes
- iTLB (instruction TLB): caches translations for instruction fetches
A TLB miss means the translation wasn’t cached. The CPU then has to walk the page table, which is a multi-level lookup and takes a few dozen cycles. On a TLB miss, the CPU stalls until the translation is available.
PMC events for TLB misses
| Metric | Event | Umask | Intel SDM Name |
|---|---|---|---|
| dTLB load miss (causes walk) | 0x08 | 0x01 | DTLB_LOAD_MISSES.MISS_CAUSES_A_WALK |
| dTLB load walk completed | 0x08 | 0x02 | DTLB_LOAD_MISSES.WALK_COMPLETED |
| iTLB miss (causes walk) | 0x85 | 0x01 | ITLB_MISSES.MISS_CAUSES_A_WALK |
| iTLB walk completed | 0x85 | 0x02 | ITLB_MISSES.WALK_COMPLETED |
Notice that cache and TLB events are in different event families. Cache events are 0xD1 (MEM_LOAD_RETIRED). dTLB events are 0x08 (DTLB_LOAD_MISSES). iTLB events are 0x85 (ITLB_MISSES). The “causes a walk” events are the useful ones — a TLB miss that doesn’t trigger a page table walk (e.g., a hit in the second-level TLB) isn’t as costly, so we count the ones that actually stall the core waiting for a translation.
The event numbers above are Intel-specific. AMD Zen uses different events and different event numbers for TLB misses — the cross-microarchitecture table below covers that. On AMD, the closest equivalent of Intel’s “iTLB miss that causes a page table walk” is event 0x84 (BpL1TlbMissL2Miss), not event 0x85. Event 0x85 on AMD (BpL1TlbMissL2Hit) counts L1 ITLB misses that hit in the L2 ITLB — these are cheap and don’t cause a walk. The costly misses are the ones that miss both L1 and L2, which is event 0x84.
A function to open cache counters
#![allow(unused)]
fn main() {
// PERF_TYPE_RAW = 4: raw PMC events (event numbers vary by CPU)
const PERF_TYPE_RAW: u32 = 4;
fn open_raw_pmc(
event: u16, // event number from Intel SDM
umask: u8, // unit mask from Intel SDM
pid: libc::pid_t,
cpu: libc::c_int,
) -> std::io::Result<libc::c_int> {
// Hand-rolled perf_event_attr — libc::perf_event_attr isn't available
// on all platforms. The field ordering must match the kernel struct exactly.
#[repr(C)]
struct PerfEventAttr {
type_: u32,
size: u32,
config: u64,
sample_period: u64,
sample_type: u64,
read_format: u64,
flags: u64, // bit 0: disabled, bit 2: pinned
}
let attr = PerfEventAttr {
type_: PERF_TYPE_RAW,
size: std::mem::size_of::<PerfEventAttr>() as u32,
config: ((umask as u64) << 8) | (event as u64),
sample_period: 0, // counting mode: no sampling, read counter directly
sample_type: 0,
read_format: 0,
flags: 0b101, // disabled=1 (bit 0), pinned=1 (bit 2)
};
let fd = unsafe {
libc::syscall(
libc::SYS_perf_event_open,
&attr as *const _,
pid,
cpu,
-1,
0,
)
};
if fd < 0 {
return Err(std::io::Error::last_os_error());
}
Ok(fd as libc::c_int)
}
fn enable_counter(fd: libc::c_int) -> std::io::Result<()> {
let ret = unsafe {
// arg=0 enables this counter (not a group)
libc::ioctl(fd, libc::PERF_EVENT_IOC_ENABLE, 0)
};
if ret < 0 {
return Err(std::io::Error::last_os_error());
}
Ok(())
}
}Now opening specific counters:
#![allow(unused)]
fn main() {
// Open a counter for L1 dcache load miss (event 0xD1, umask 0x08)
// pid=0: measure this process; cpu=-1: follow across all CPUs
let l1d_miss = open_raw_pmc(0xD1, 0x08, 0, -1)?;
// Open a counter for dTLB load miss (event 0x08, umask 0x01)
let dtlb_miss = open_raw_pmc(0x08, 0x01, 0, -1)?;
// Open a counter for L3 cache miss (event 0xD1, umask 0x20)
let l3_miss = open_raw_pmc(0xD1, 0x20, 0, -1)?;
}If a counter returns EINVAL, that event isn’t available on this CPU. Catch it and fall back.
System-wide monitoring: The examples above use
pid=0(measure the calling process) for simplicity. A production monitoring tool should usepid=-1with per-CPU file descriptors to measure all processes on the system. See Part 3’s “Per-process vs. system-wide monitoring” section for the full explanation.
Computing miss rates
Raw counts don’t mean much in isolation. A workload doing a billion memory accesses might have 50 million L3 misses. Is that a lot? It depends on how many accesses there were. The useful metric is miss rate per thousand instructions or miss rate per million memory operations.
#![allow(unused)]
fn main() {
fn compute_miss_rate(misses: u64, instructions: u64) -> f64 {
if instructions == 0 {
return 0.0;
}
(misses as f64 / instructions as f64) * 1000.0
}
}A few reference points for context:
- L1 dcache miss rate: 2-5 misses per 1000 instructions is typical for well-optimized code
- L3 miss rate: 0.5-2 per 1000 instructions means memory locality is decent; above 5 suggests poor spatial locality
- dTLB miss rate: 0.1-1 per 1000 is typical; above 5 suggests TLB-unfriendly access patterns (e.g., scanning large arrays with a large page table stride)
Cross-microarchitecture event selection
Here’s a wrapper that selects the right event numbers based on the detected microarchitecture:
#![allow(unused)]
fn main() {
use crate::cpu::{detect, CpuMicroarch, Microarch};
#[derive(Clone, Copy)]
pub struct CacheEvent {
pub name: &'static str,
pub event: u16, // event number from Intel SDM
pub umask: u8, // unit mask from Intel SDM
}
fn cache_events_for(microarch: &Microarch) -> Vec<CacheEvent> {
match microarch {
Microarch::Skylake | Microarch::SkylakeSp | Microarch::KabyLake => vec![
CacheEvent { name: "L1 dcache miss", event: 0xD1, umask: 0x08 },
CacheEvent { name: "L2 cache miss", event: 0xD1, umask: 0x10 },
CacheEvent { name: "L3 cache miss", event: 0xD1, umask: 0x20 },
CacheEvent { name: "dTLB load miss", event: 0x08, umask: 0x01 },
CacheEvent { name: "iTLB miss", event: 0x85, umask: 0x01 },
],
Microarch::IceLake | Microarch::IceLakeSp | Microarch::RocketLake
| Microarch::SapphireRapids => vec![
// Ice Lake changed some event encodings.
// L3 miss (0xD1, umask 0x20) is not reliably available on all Ice Lake SKUs.
// Check `perf list` on your system; if MEM_LOAD_RETIRED.L3_MISS is listed,
// add it with event 0xD1, umask 0x20.
CacheEvent { name: "L1 dcache miss", event: 0xD1, umask: 0x08 },
CacheEvent { name: "L2 cache miss", event: 0xD1, umask: 0x10 },
CacheEvent { name: "dTLB load miss", event: 0x08, umask: 0x01 },
CacheEvent { name: "iTLB miss", event: 0x85, umask: 0x01 },
],
Microarch::AlderLake => vec![
// Alder Lake P-cores (Golden Cove) use Ice Lake PMC encodings;
// E-cores (Gracemont) have different encodings. This table
// returns the P-core events. A production tool would need to
// detect the core type (CPUID leaf 0x1A) and select per-core.
// The L3 miss counter is not reliably available on all SKUs
// (same as Ice Lake) — check `perf list`.
CacheEvent { name: "L1 dcache miss", event: 0xD1, umask: 0x08 },
CacheEvent { name: "L2 cache miss", event: 0xD1, umask: 0x10 },
CacheEvent { name: "dTLB load miss", event: 0x08, umask: 0x01 },
CacheEvent { name: "iTLB miss", event: 0x85, umask: 0x01 },
],
// AMD Zen — completely different event encodings from Intel.
// AMD event numbers from the Linux kernel's amd_hw_cache_event_ids
// table (arch/x86/events/amd/core.c) and the AMD Processor
// Programming Reference (PPR) for each Zen generation.
//
// AMD doesn't expose a direct L3 miss counter in the same way as
// Intel. L2 miss (event 0x7E) is the closest proxy — it counts
// accesses that missed the L2, which includes L3 misses. For a
// more precise L3 miss rate on Zen, use IBS (Instruction-Based
// Sampling) or the L3_perf events documented in the AMD PPR.
Microarch::Zen | Microarch::ZenPlus | Microarch::Zen2 => vec![
CacheEvent { name: "L1 dcache miss", event: 0x41, umask: 0x01 }, // Data Cache Misses
CacheEvent { name: "L2 cache miss", event: 0x7E, umask: 0x03 }, // L2 Misses (IC+DC)
CacheEvent { name: "dTLB load miss", event: 0x45, umask: 0x0F }, // DTLB Misses (all page sizes)
CacheEvent { name: "iTLB miss", event: 0x84, umask: 0x01 }, // L1 ITLB miss + L2 ITLB miss (page table walk)
],
Microarch::Zen3 | Microarch::Zen4 | Microarch::Zen5 => vec![
// Zen 3+ reuses the same core PMC event numbers for cache/TLB.
// The L2 miss event changed slightly in Zen 3 — event 0x7E
// still works but the recommended event for L3 miss proxy is
// 0x7E with umask 0x28 (L2 misses that miss L3 too). Check
// the PPR for your specific model.
CacheEvent { name: "L1 dcache miss", event: 0x41, umask: 0x01 },
CacheEvent { name: "L2 cache miss", event: 0x7E, umask: 0x03 },
CacheEvent { name: "dTLB load miss", event: 0x45, umask: 0x0F },
CacheEvent { name: "iTLB miss", event: 0x84, umask: 0x01 }, // L1 ITLB miss + L2 ITLB miss
],
_ => {
// Unknown microarchitecture — fall back to the universal
// hardware events (instructions and cycles, no cache events)
// since raw events are microarchitecture-specific and guessing
// wrong gives silently incorrect counts.
vec![]
}
}
}
}Counting mode vs. sampling mode
We’ve been using counting mode: open a counter, enable it, and read the cumulative count every second. This gives you a metric — a number that describes what’s happening.
Sampling mode is different: you set the counter to generate a sample (a trace event) every N events. The kernel writes each sample to a ring buffer that userspace reads. This gives you a profile — a stream of individual events that lets you see where the misses are happening.
For a monitoring dashboard, counting mode is what you want. You’re interested in “how many L3 misses per second” — not “which function is causing L3 misses.”
The performance monitoring landscape: counting mode gives metrics, sampling mode gives profiles. We’re building a metrics collector, so counting is the right tool.
Next: Part 6 — Scheduler Tracing with eBPF — Instrument the scheduler with tracepoints and measure runqueue wait times.
Part 6 — Scheduler Tracing with eBPF
The scheduler decides which task runs next. Every scheduling decision is a data point.
You can’t see this from PMCs. The CPU doesn’t know whether it’s executing a task that’s been waiting for 50 milliseconds or one that recently got scheduled. The kernel knows. It records every scheduling decision in tracepoints, and we can read those tracepoints with eBPF.
The scheduler tracepoints
The kernel exposes several scheduler tracepoints. The ones we care about:
| Tracepoint | When it fires |
|---|---|
sched:sched_switch | The scheduler switches from one task to another |
sched:sched_waking | A task is about to be woken (pre-wakeup) |
sched:sched_wakeup | A sleeping task has been woken |
sched:sched_stat_wait | Time a task spent waiting on a runqueue |
sched:sched_migrate_task | A task was migrated to another CPU |
sched_switch is the most informative. It fires whenever the scheduler replaces the running task with a different one.
The map declaration pattern
This is the most important pattern in eBPF programming with Aya: maps are static globals.
You declare a map as a static variable with a constructor call. The eBPF verifier sees it at load time and allocates it. You reference it by name from userspace.
#![allow(unused)]
fn main() {
use aya_ebpf::maps::RingBuf;
use aya_ebpf::macros::map;
// 8 KiB ring buffer. Must be a power-of-two multiple of page_size (4096).
// The userspace reader picks this up by the name "events".
#[map]
static EVENTS: RingBuf = RingBuf::with_byte_size(8 * 4096, 0);
#[map]
static COUNTERS: aya_ebpf::maps::HashMap<u32, u64> =
aya_ebpf::maps::HashMap::with_max_entries(256, 0);
}Note that aya-ebpf maps don’t have builder methods — they’re constructed with with_max_entries() and with_byte_size() (for ring buffers). This is different from Rust’s standard library conventions, but it matches how the eBPF verifier needs to know the map size at compile time.
Reading tracepoint arguments
A tracepoint has a fixed payload — a chunk of memory that contains the arguments. The kernel defines the format. You read from the tracepoint context at specific byte offsets using read_at().
Here’s the layout for sched:sched_switch on Linux 5.x and 6.x:
Offset Type Field
------ ---- -----
0 char[16] prev_comm (TASK_COMM_LEN)
16 u32 prev_pid
20 u32 prev_prio
24 u64 prev_state (the TASK_* state mask)
32 char[16] next_comm (TASK_COMM_LEN)
48 u32 next_pid
52 u32 next_prio
The prev_comm and next_comm fields are 16-byte character arrays containing the process name (TASK_COMM_LEN is 16 in the kernel). They take up space in the payload even though we don’t read them — every offset after them is shifted.
PID types:
u32vsi32. The kernel defines PIDs aspid_t, which isint(i32on all supported platforms). We readprev_pidandnext_pidasu32here because the struct fields are unsigned and we only need the> 0check. In thesched_wakinghandler below, we readpidasi32because theWAKE_TShash map usesi32keys — matchingpid_t. Thenext_pid as i32cast in the combined handler bridges the two. Both types work for positive PIDs; the cast is zero-cost at runtime.
Always verify on your system:
cat /sys/kernel/tracing/events/sched/sched_switch/format
This prints the exact field layout including offsets. The format file offsets include the 8-byte common header, so subtract 8 to get the read_at offset. See the detailed cross-check procedure in Part 2. Kernel versions can and do change tracepoint layouts.
The struct in our eBPF program:
#![allow(unused)]
fn main() {
// ebpf-programs/src/scheduler.rs
use aya_ebpf::programs::TracePointContext;
use aya_ebpf::macros::tracepoint;
use aya_ebpf::maps::RingBuf;
use aya_ebpf::helpers::{bpf_ktime_get_ns, bpf_get_smp_processor_id};
#[derive(Clone, Copy)]
#[repr(C)]
pub struct SchedSwitchEvent {
pub cpu_id: u32,
pub prev_pid: u32,
pub prev_state: u64,
pub next_pid: u32,
pub timestamp: u64,
}
// Declare the ring buffer as a static
#[map]
static EVENTS: RingBuf = RingBuf::with_byte_size(8 * 4096, 0);
#[tracepoint]
pub fn sched_switch(ctx: TracePointContext) -> u32 {
let prev_pid = unsafe { ctx.read_at::<u32>(16).unwrap_or(0) };
let prev_state = unsafe { ctx.read_at::<u64>(24).unwrap_or(0) };
let next_pid = unsafe { ctx.read_at::<u32>(48).unwrap_or(0) };
let cpu_id = unsafe { bpf_get_smp_processor_id() };
let timestamp = unsafe { bpf_ktime_get_ns() };
let event = SchedSwitchEvent {
cpu_id,
prev_pid,
prev_state,
next_pid,
timestamp,
};
// output() sends data directly to the ring buffer
EVENTS.output(&event, 0);
0
}
}A few things to notice here:
unsafe around read_at: read_at wraps bpf_probe_read, which reads from kernel memory. The eBPF verifier can’t guarantee the memory is valid, so you need unsafe. In practice, you’re reading from a tracepoint payload that the kernel has placed there, so it’s safe — but the compiler doesn’t know that.
bpf_get_smp_processor_id() and bpf_ktime_get_ns(): These are raw BPF helpers. Aya wraps many helpers, but these two are so fundamental that they’re exposed directly as unsafe extern calls through the bindings. They’re available in every eBPF program.
EVENTS.output(): The ring buffer is a static. We call .output() directly on it — no ctx involved. The ring buffer is declared at the top of the file and lives for the lifetime of the program.
Building a runqueue wait histogram
The wait time is the time between when a task was woken and when it actually starts running on a CPU. To measure it, we need to correlate events from two tracepoints: sched:sched_waking (when a task is about to wake) and sched:sched_switch (when a task starts running). When sched_switch fires and the next_pid matches a task we saw in sched_waking, we know how long that task waited.
The approach: a hash map keyed by PID. When a task is woken (via sched_waking), record the timestamp. When a task starts running (via sched_switch where next_pid matches), look up the timestamp, compute the wait, and delete the entry.
Why sched_waking instead of sched_wakeup? sched_waking fires when the wake signal is about to be sent — slightly earlier and more reliable for measuring the full wait. sched_wakeup fires after the target task has been added to the runqueue. For wait time measurement, the difference is negligible, but sched_waking is the more common choice in production tools.
The sched:sched_waking payload layout:
Offset Type Field
------ ---- -----
0 char[16] comm (TASK_COMM_LEN)
16 i32 pid (pid_t — signed)
20 i32 prio
24 i32 target_cpu
Verify on your system: cat /sys/kernel/tracing/events/sched/sched_waking/format (format file offsets are 8 bytes larger than read_at offsets — see Part 2 for details)
#![allow(unused)]
fn main() {
// ebpf-programs/src/scheduler.rs
use aya_ebpf::programs::TracePointContext;
use aya_ebpf::macros::{map, tracepoint};
use aya_ebpf::maps::{HashMap, RingBuf};
use aya_ebpf::helpers::{bpf_ktime_get_ns, bpf_get_smp_processor_id};
// Map: PID → waking timestamp (nanoseconds)
#[map]
static WAKE_TS: HashMap<i32, u64> = HashMap::with_max_entries(1024, 0);
// Ring buffer for wait events
#[map]
static EVENTS: RingBuf = RingBuf::with_byte_size(8 * 4096, 0);
#[derive(Clone, Copy)]
#[repr(C)]
pub struct WaitEvent {
pub pid: u32,
pub wait_ns: u64,
pub cpu_id: u32,
}
// sched:sched_waking — record when a task is about to be woken
// Payload: comm (char[16]) at 0, pid (i32) at 16, prio (i32) at 20, target_cpu (i32) at 24
#[tracepoint]
pub fn sched_waking(ctx: TracePointContext) -> u32 {
let pid = unsafe { ctx.read_at::<i32>(16).unwrap_or(0) };
let ts = unsafe { bpf_ktime_get_ns() };
// Only record for non-zero PIDs (kernel threads have pid 0)
if pid > 0 {
WAKE_TS.insert(&pid, &ts, 0);
}
0
}
// sched:sched_switch — check if the incoming task was waiting
// Payload: prev_comm (char[16]) at 0, prev_pid (u32) at 16, prev_prio (u32) at 20,
// prev_state (u64) at 24, next_comm (char[16]) at 32, next_pid (u32) at 48
#[tracepoint]
pub fn sched_switch_wait(ctx: TracePointContext) -> u32 {
let next_pid = unsafe { ctx.read_at::<u32>(48).unwrap_or(0) };
if next_pid > 0 {
let ts = unsafe { bpf_ktime_get_ns() };
// WAKE_TS key type is i32 (kernel PIDs are pid_t). Cast for lookup.
// Safe because we only reach this branch when next_pid > 0,
// and all real PIDs fit in both u32 and i32.
let pid_key = next_pid as i32;
// SAFETY: WAKE_TS.get() is unsafe because the kernel doesn't
// guarantee atomicity without BPF_F_NO_PREALLOC. For our purposes —
// measuring scheduler wait time — occasional corruption is acceptable
// since it means one lost measurement at worst.
unsafe {
if let Some(&wake_ts) = WAKE_TS.get(&pid_key) {
let wait_ns = ts.saturating_sub(wake_ts);
let cpu_id = bpf_get_smp_processor_id();
let event = WaitEvent { pid: next_pid, wait_ns, cpu_id };
EVENTS.output(&event, 0);
let _ = WAKE_TS.remove(&pid_key);
}
}
}
0
}
}Two tracepoints, one hash map. sched_waking writes the timestamp when a task is about to wake. sched_switch reads it back when the task starts running.
Merging the handlers: This chapter shows three separate sched_switch handlers — one for basic tracing, one for wait histograms, one for involuntary switches. In a real program, you’d combine them into a single handler. They’re split here so each concept is clear on its own. When you merge them, the combined handler reads prev_pid, prev_state, and next_pid once, then does all three operations (ring buffer output, wait lookup, involuntary counting) in the same function. The next_pid in sched_switch matches the pid from sched_waking — that’s how we correlate the two events. The wait time is the difference between the two timestamps.
This is harder than a single-tracepoint approach — you need to correlate events from two sources. But it’s the only correct way to measure runqueue wait time with the tracepoints the kernel actually provides. The scheduler’s internal enqueue/dequeue operations aren’t exposed as tracepoints.
insert(), get(), remove(): HashMap in Aya eBPF has insert() (returns Result<(), c_long>), get() (unsafe, returns Option<&V>), and remove() (returns Result<(), c_long>). There’s no get_mut() — eBPF maps are accessed by reference.
get() returns Option<&V>: When the key isn’t found, get() returns None. Two patterns for handling this:
if let Some(&val) = map.get(&key)— when the “not found” case means “skip this event.” Used insched_switch_waitabove: if the PID isn’t inWAKE_TS, there’s nothing to compute.map.get(&key).copied().unwrap_or(0)— when the “not found” case means “start from zero.” Used for counter increments: if the CPU isn’t in the map, the count is zero.
Both patterns are idiomatic. Pick based on the semantics of the “not found” case.
Counting context switches
Every sched_switch is a context switch. Counting them is straightforward — we’ll extend the sched_switch handler from the previous section with a per-CPU hash map:
#![allow(unused)]
fn main() {
// ebpf-programs/src/scheduler.rs
#[map]
static CTXSW_COUNTERS: HashMap<u32, u64> = HashMap::with_max_entries(256, 0);
#[tracepoint]
pub fn sched_switch_count(ctx: TracePointContext) -> u32 {
let cpu_id = unsafe { bpf_get_smp_processor_id() };
unsafe {
let count = CTXSW_COUNTERS.get(&cpu_id).copied().unwrap_or(0u64);
let new_count = count + 1;
let _ = CTXSW_COUNTERS.insert(&cpu_id, &new_count, 0);
}
0
}
}Note that insert() takes ownership of the reference’s pointed-to value, so we dereference count into new_count and insert that. This is the standard pattern for counter increments in eBPF.
Involuntary context switches
An involuntary context switch is one where the running task didn’t voluntarily give up the CPU — it got preempted or its time slice expired. We can detect this from the same sched_switch tracepoint by checking whether prev_pid was actually running when it got switched out.
In the kernel, TASK_RUNNING is state 0. If prev_state in sched_switch is 0, the task was running and got switched out involuntarily:
#![allow(unused)]
fn main() {
// TASK_RUNNING = 0 in the kernel
const TASK_RUNNING: u64 = 0;
#[map]
static INVOL_CTXSW: HashMap<u32, u64> = HashMap::with_max_entries(256, 0);
#[tracepoint]
pub fn sched_switch_involuntary(ctx: TracePointContext) -> u32 {
let prev_state = unsafe { ctx.read_at::<u64>(24).unwrap_or(0) };
let prev_pid = unsafe { ctx.read_at::<u32>(16).unwrap_or(0) };
// prev_state == 0 means the task was in TASK_RUNNING
// prev_pid > 0 means it wasn't idle — involuntary switch
let is_involuntary = prev_state == TASK_RUNNING && prev_pid > 0;
if is_involuntary {
let cpu = unsafe { bpf_get_smp_processor_id() };
unsafe {
let count = INVOL_CTXSW.get(&cpu).copied().unwrap_or(0u64);
let new_count = count + 1;
let _ = INVOL_CTXSW.insert(&cpu, &new_count, 0);
}
}
0
}
}Steal time
On virtualized systems, the hypervisor sometimes doesn’t give a vCPU any time to run even though it was runnable. The kernel records this as steal time. Unlike other metrics in this tutorial, steal time isn’t available through a scheduler tracepoint — it’s reported in /proc/stat:
cpu 2255 34 2290 22625563 6290 0 236 0 0 0
cpu0 1132 17 1145 11312781 3145 0 118 0 0 0
cpu1 1123 17 1145 11312782 3145 0 118 0 0 0
The fields are: user, nice, system, idle, iowait, irq, softirq, steal, guest, guest_nice. The steal field (8th column, 0-indexed column 7) is the number of jiffies the CPU wanted to run but the hypervisor scheduled something else.
#![allow(unused)]
fn main() {
use std::fs;
struct CpuStat {
pub user: u64,
pub nice: u64,
pub system: u64,
pub idle: u64,
pub iowait: u64,
pub irq: u64,
pub softirq: u64,
pub steal: u64,
}
fn read_proc_stat() -> std::io::Result<Vec<CpuStat>> {
let content = fs::read_to_string("/proc/stat")?;
let mut cpus = Vec::new();
for line in content.lines() {
let parts: Vec<&str> = line.split_whitespace().collect();
if parts.is_empty() || !parts[0].starts_with("cpu") {
continue;
}
// Skip the aggregate "cpu " line — we want per-CPU (cpu0, cpu1, ...)
if parts[0] == "cpu" {
continue;
}
cpus.push(CpuStat {
user: parts.get(1).and_then(|v| v.parse().ok()).unwrap_or(0),
nice: parts.get(2).and_then(|v| v.parse().ok()).unwrap_or(0),
system: parts.get(3).and_then(|v| v.parse().ok()).unwrap_or(0),
idle: parts.get(4).and_then(|v| v.parse().ok()).unwrap_or(0),
iowait: parts.get(5).and_then(|v| v.parse().ok()).unwrap_or(0),
irq: parts.get(6).and_then(|v| v.parse().ok()).unwrap_or(0),
softirq:parts.get(7).and_then(|v| v.parse().ok()).unwrap_or(0),
steal: parts.get(8).and_then(|v| v.parse().ok()).unwrap_or(0),
});
}
Ok(cpus)
}
}Compute the steal ratio (fraction of total CPU time spent stolen):
#![allow(unused)]
fn main() {
fn steal_ratio(stat: &CpuStat) -> f64 {
let total = stat.user + stat.nice + stat.system + stat.idle
+ stat.iowait + stat.irq + stat.softirq + stat.steal;
if total == 0 {
return 0.0;
}
stat.steal as f64 / total as f64
}
}Steal time above 5-10% means the host is oversubscribed — there are more vCPUs competing for physical CPUs than physical CPUs available. Your workload is spending real time waiting for the hypervisor, not doing useful work.
This is a procfs metric, not an eBPF metric — no tracepoint or kprobe required. The kernel already tracks it. We read it alongside the scheduler tracepoint data in the same polling loop.
Checking tracepoint availability. Scheduler tracepoints are available on virtually every Linux kernel, but their names and arguments can change between versions. Before your monitoring tool starts, verify that the tracepoints it needs actually exist:
# List all scheduler tracepoints on this kernel ls /sys/kernel/tracing/events/sched/ # Check a specific tracepoint cat /sys/kernel/tracing/events/sched/sched_switch/idIf
cat .../idprints a number, the tracepoint exists and can be attached. If you get “No such file or directory,” the tracepoint isn’t available on this kernel — your program should skip attaching it rather than crashing. In the userspace code below, you’d guard theprogram.attach()call with a check like this:#![allow(unused)] fn main() { // Check that the tracepoint exists before attaching let tp_id = std::fs::read_to_string( "/sys/kernel/tracing/events/sched/sched_switch/id" ); if tp_id.is_ok() { program.attach("sched", "sched_switch")?; } else { eprintln!("sched:sched_switch not available on this kernel, skipping"); } }
Merging the handlers
The three sched_switch handlers above each do one thing. In a real program, you’d combine them into a single handler that reads the tracepoint payload once and does all three operations. Here’s what that looks like:
#![allow(unused)]
fn main() {
// ebpf-programs/src/scheduler.rs
use aya_ebpf::programs::TracePointContext;
use aya_ebpf::macros::{map, tracepoint};
use aya_ebpf::maps::{HashMap, RingBuf};
use aya_ebpf::helpers::{bpf_ktime_get_ns, bpf_get_smp_processor_id};
const TASK_RUNNING: u64 = 0;
// Shared maps (declared once, used by both sched_switch and sched_waking)
#[map]
static EVENTS: RingBuf = RingBuf::with_byte_size(8 * 4096, 0);
#[map]
static WAKE_TS: HashMap<i32, u64> = HashMap::with_max_entries(1024, 0);
#[map]
static CTXSW_COUNTERS: HashMap<u32, u64> = HashMap::with_max_entries(256, 0);
#[map]
static INVOL_CTXSW: HashMap<u32, u64> = HashMap::with_max_entries(256, 0);
#[derive(Clone, Copy)]
#[repr(C)]
pub struct SchedSwitchEvent {
pub cpu_id: u32,
pub prev_pid: u32,
pub prev_state: u64,
pub next_pid: u32,
pub wait_ns: u64, // 0 if no waking timestamp was found
pub timestamp: u64,
}
// Combined sched_switch handler: tracing + wait lookup + involuntary counting
//
// Payload layout (Linux 5.x/6.x — verify with
// cat /sys/kernel/tracing/events/sched/sched_switch/format):
// offset 0: prev_comm char[16]
// offset 16: prev_pid u32
// offset 20: prev_prio u32
// offset 24: prev_state u64
// offset 32: next_comm char[16]
// offset 48: next_pid u32
// offset 52: next_prio u32
#[tracepoint]
pub fn sched_switch_combined(ctx: TracePointContext) -> u32 {
// Read the payload once — shared across all three operations
let prev_pid = unsafe { ctx.read_at::<u32>(16).unwrap_or(0) };
let prev_state = unsafe { ctx.read_at::<u64>(24).unwrap_or(0) };
let next_pid = unsafe { ctx.read_at::<u32>(48).unwrap_or(0) };
let cpu_id = unsafe { bpf_get_smp_processor_id() };
let timestamp = unsafe { bpf_ktime_get_ns() };
// --- Operation 1: Ring buffer event ---
// Look up the waking timestamp for next_pid to compute wait time
let mut wait_ns = 0u64;
if next_pid > 0 {
let pid_key = next_pid as i32;
unsafe {
if let Some(&wake_ts) = WAKE_TS.get(&pid_key) {
wait_ns = timestamp.saturating_sub(wake_ts);
let _ = WAKE_TS.remove(&pid_key);
}
}
}
let event = SchedSwitchEvent {
cpu_id,
prev_pid,
prev_state,
next_pid,
wait_ns,
timestamp,
};
EVENTS.output(&event, 0);
// --- Operation 2: Context switch counter ---
unsafe {
let count = CTXSW_COUNTERS.get(&cpu_id).copied().unwrap_or(0u64);
let _ = CTXSW_COUNTERS.insert(&cpu_id, &(count + 1), 0);
}
// --- Operation 3: Involuntary context switch counter ---
if prev_state == TASK_RUNNING && prev_pid > 0 {
unsafe {
let count = INVOL_CTXSW.get(&cpu_id).copied().unwrap_or(0u64);
let _ = INVOL_CTXSW.insert(&cpu_id, &(count + 1), 0);
}
}
0
}
}Key differences from the separate handlers:
- One
read_atpass. The combined handler readsprev_pid,prev_state, andnext_pidonce. The separate handlers each read independently — that’s three times the work per context switch. - Wait time is embedded in the event. The
SchedSwitchEventstruct now includeswait_ns. If the incoming task (next_pid) doesn’t have a waking timestamp,wait_nsis 0 — the userspace reader checks for this. - One
sched_wakinghandler still needed. The combinedsched_switchhandler only replaces the three separatesched_switchvariants. Thesched_wakinghandler (which records the wake timestamp) is unchanged.
The lib.rs registration for the combined approach needs only two entry points instead of three:
#![allow(unused)]
fn main() {
// ebpf-programs/src/lib.rs
#![no_std]
use aya_ebpf::macros::tracepoint;
use aya_ebpf::programs::TracePointContext;
mod scheduler;
#[tracepoint]
pub fn trace_sched_switch(ctx: TracePointContext) -> u32 {
scheduler::sched_switch_combined(ctx)
}
#[tracepoint]
pub fn trace_sched_waking(ctx: TracePointContext) -> u32 {
scheduler::sched_waking(ctx)
}
}Reading in userspace
The userspace side loads the eBPF programs, creates maps, attaches tracepoints, and reads from the ring buffer. The struct below matches the combined handler — it includes wait_ns. If you use one of the separate handlers instead, omit wait_ns from the struct.
#![allow(unused)]
fn main() {
// monitor/src/main.rs
use aya::maps::RingBuf;
use aya::programs::TracePoint;
use aya::Ebpf;
use std::convert::TryFrom;
#[derive(Clone, Copy, Debug)]
#[repr(C)]
pub struct SchedSwitchEvent {
pub cpu_id: u32,
pub prev_pid: u32,
pub prev_state: u64,
pub next_pid: u32,
pub wait_ns: u64, // 0 if no waking timestamp was found (combined handler)
pub timestamp: u64,
}
async fn poll_scheduler(ebpf: &mut Ebpf) -> anyhow::Result<()> {
// RingBuf takes ownership of the map reference
let mut ring_buf = RingBuf::try_from(ebpf.map_mut("events")?)?;
let mut ctxsw_total = 0u64;
let mut involuntary = 0u64;
while let Some(item) = ring_buf.next() {
// item derefs to &[u8] — cast to our event type
let event = unsafe {
std::ptr::read_unaligned(item.as_ptr() as *const SchedSwitchEvent)
};
ctxsw_total += 1;
// Involuntary: was running (prev_state == 0) and wasn't idle (prev_pid > 0)
if event.prev_state == 0 && event.prev_pid > 0 {
involuntary += 1;
}
}
if ctxsw_total > 0 {
let inv_rate = involuntary as f64 / ctxsw_total as f64;
println!("ctxsw={ctxsw_total} involuntary_rate={inv_rate:.3}");
}
Ok(())
}
}Key points about the userspace code:
RingBuf::try_from(map_data): You create the ring buffer from a map reference. ebpf.map_mut("events") looks up the map by name that the eBPF program declared. RingBuf::try_from() takes ownership of that map reference.
ring_buf.next() returns Option<RingBufItem<'_>>: RingBufItem derefs to &[u8] — a byte slice containing the event data. You cast it back to your struct type. read_unaligned is important because the data may not be aligned to the struct’s alignment requirements.
TracePoint (capital P): The userspace program type is TracePoint, not Tracepoint. This is the Aya convention — program types are PascalCase.
Reading histogram maps
For counter maps (like CTXSW_COUNTERS and INVOL_CTXSW), you read them directly from userspace without going through the ring buffer:
#![allow(unused)]
fn main() {
use aya::maps::HashMap;
async fn read_counters(ebpf: &mut Ebpf) -> anyhow::Result<()> {
let counters: HashMap<_, u32, u64> =
HashMap::try_from(ebpf.map_mut("ctxsw_counters")?)?;
for (cpu, &count) in counters.iter() {
println!("cpu={cpu} ctxsw={count}");
}
Ok(())
}
}The counter values are cumulative since the program was loaded. To get per-second rates, save the previous reading and compute the delta.
Registering programs in lib.rs
The eBPF programs are defined in separate .rs files and registered in lib.rs:
#![allow(unused)]
fn main() {
// ebpf-programs/src/lib.rs
#![no_std]
use aya_ebpf::macros::tracepoint;
use aya_ebpf::programs::TracePointContext;
mod scheduler;
#[tracepoint]
pub fn trace_sched_switch(ctx: TracePointContext) -> u32 {
scheduler::sched_switch(ctx)
}
#[tracepoint]
pub fn trace_sched_waking(ctx: TracePointContext) -> u32 {
scheduler::sched_waking(ctx)
}
#[tracepoint]
pub fn trace_sched_switch_wait(ctx: TracePointContext) -> u32 {
scheduler::sched_switch_wait(ctx)
}
#[tracepoint]
pub fn trace_sched_switch_count(ctx: TracePointContext) -> u32 {
scheduler::sched_switch_count(ctx)
}
#[tracepoint]
pub fn trace_sched_switch_involuntary(ctx: TracePointContext) -> u32 {
scheduler::sched_switch_involuntary(ctx)
}
}Each entry point uses the #[tracepoint] form — no args. The macro marks the function as an eBPF tracepoint program; the category and name are provided from userspace via program.attach(). The function body delegates to the module implementation. In a real project, you might define the programs directly in main.rs or split them into modules — either way, the macro goes on the function that the eBPF verifier sees.
Next: Part 7 — NUMA and Memory Metrics — Track page migration rates and remote memory access ratios on multi-socket systems.
Part 7 — NUMA and Memory Metrics
On a NUMA system, memory access has a cost that depends on which socket you’re on.
NUMA stands for Non-Uniform Memory Access. On a multi-socket server, each CPU socket has its own local memory. When a task running on socket A accesses memory attached to socket B, the data has to travel across the inter-socket interconnect. That takes longer than accessing local memory.
If your workload is hitting 80% remote memory access, you’re paying the interconnect tax on most of your memory traffic. That’s a NUMA problem.
The basics in plain language
A node in Linux’s NUMA vocabulary is a group of CPUs and memory that are physically close. On a dual-socket system, you typically have node 0 and node 1. Each node has its own local memory. Memory attached to node 0 is “local” to socket 0’s CPUs, and “remote” to socket 1’s CPUs.
Linux has a NUMA balancer that moves pages between nodes at runtime to try to keep tasks running close to their data. When the balancer kicks in, it migrates pages. Too much migration is a sign that tasks are bouncing between sockets.
/proc/vmstat — the key file
/proc/vmstat is a flat list of virtual memory statistics. Most of them are for kernel internals, but a few are relevant for NUMA:
numa_hit 12345678 // pages allocated to this node (success)
numa_miss 234567 // pages allocated to this node but from remote (fail)
numa_foreign 12345 // pages allocated to this node from another node's memory
pgmigrate_success 98765 // pages successfully migrated between nodes
pgmigrate_fail 123 // migration attempts that failed
The key fields:
numa_hit: pages that were allocated on this node and used on this node (the good case)numa_miss: pages allocated on this node but the CPU accessing them was on a remote node (the bad case)pgmigrate_success: how many pages were successfully moved between nodes
The NUMA remote ratio is:
remote_ratio = numa_miss / (numa_hit + numa_miss)
A remote ratio above 0.2 (20%) means more than a fifth of memory accesses are crossing the interconnect. This is worth optimizing.
numa_foreign is the mirror of numa_miss — it counts pages that were allocated on this node but accessed from a remote node. Where numa_miss tells you how many remote accesses this node suffered, numa_foreign tells you how many local pages were stolen by remote CPUs. On a two-socket system, numa_miss on node 0 roughly equals numa_foreign on node 1 (and vice versa), because a remote access on one side is a foreign allocation on the other. The two numbers give you both perspectives: numa_miss shows the performance impact (your CPU is waiting for remote memory), numa_foreign shows the cause (some other CPU is pulling your data across the interconnect).
/sys/devices/system/node/nodeN/meminfo — per-node memory
Per-node memory breakdown lives in sysfs, not procfs. Each NUMA node has a meminfo file:
cat /sys/devices/system/node/node0/meminfo
The output looks like:
Node 0 Anon: 12345678 kB
Node 0 File: 2345678 kB
Node 0 HugePages: 4096 kB
Node 0 Shmem: 12345 kB
Each line breaks down memory by type:
- Anon: anonymous memory (heap, stack, not backed by a file)
- File: page cache (file-backed memory)
- HugePages: hugepages
- Shmem: shared memory (including tmpfs)
The numbers are in KiB. To get bytes: multiply by 1024.
Reading vmstat
#![allow(unused)]
fn main() {
use std::fs;
#[derive(Default)]
pub struct NumaStats {
pub numa_hit: u64,
pub numa_miss: u64,
pub numa_foreign: u64,
pub pgmigrate_success: u64,
pub pgmigrate_fail: u64,
}
fn read_vmstat() -> std::io::Result<NumaStats> {
let content = fs::read_to_string("/proc/vmstat")?;
let mut numa_hit = 0u64;
let mut numa_miss = 0u64;
let mut numa_foreign = 0u64;
let mut pgmigrate_success = 0u64;
let mut pgmigrate_fail = 0u64;
for line in content.lines() {
let mut parts = line.split_whitespace();
let name = parts.next().unwrap_or("");
let value: u64 = parts.next().unwrap_or("0").parse().unwrap_or(0);
match name {
"numa_hit" => numa_hit = value,
"numa_miss" => numa_miss = value,
"numa_foreign" => numa_foreign = value,
"pgmigrate_success" => pgmigrate_success = value,
"pgmigrate_fail" => pgmigrate_fail = value,
_ => {}
}
}
Ok(NumaStats {
numa_hit,
numa_miss,
numa_foreign,
pgmigrate_success,
pgmigrate_fail,
})
}
}Rate computation
/proc/vmstat returns cumulative counters since boot. To get per-second rates, poll the file and compute the delta:
#![allow(unused)]
fn main() {
use std::time::{Duration, Instant};
struct NumaRate {
pub numa_remote_rate: f64, // remote accesses per second
pub numa_foreign_rate: f64, // local pages accessed remotely per second
pub migration_rate: f64, // pages migrated per second
pub migration_fail_rate: f64, // failed migrations per second
}
fn compute_rates(prev: &NumaStats, curr: &NumaStats, elapsed_secs: f64) -> NumaRate {
if elapsed_secs <= 0.0 {
return NumaRate {
numa_remote_rate: 0.0,
numa_foreign_rate: 0.0,
migration_rate: 0.0,
migration_fail_rate: 0.0,
};
}
// Use saturating_sub in case counters wrap (extremely unlikely on
// real hardware, but defensive against u64 underflow on long uptimes).
let hit_delta = curr.numa_hit.saturating_sub(prev.numa_hit);
let miss_delta = curr.numa_miss.saturating_sub(prev.numa_miss);
let foreign_delta = curr.numa_foreign.saturating_sub(prev.numa_foreign);
let total_accesses = hit_delta.saturating_add(miss_delta);
let remote_accesses = miss_delta;
let migrations = curr.pgmigrate_success.saturating_sub(prev.pgmigrate_success);
let failed = curr.pgmigrate_fail.saturating_sub(prev.pgmigrate_fail);
NumaRate {
numa_remote_rate: remote_accesses as f64 / elapsed_secs,
numa_foreign_rate: foreign_delta as f64 / elapsed_secs,
migration_rate: migrations as f64 / elapsed_secs,
migration_fail_rate: failed as f64 / elapsed_secs,
}
}
}Per-node memory from sysfs
#![allow(unused)]
fn main() {
#[derive(Debug, Clone)]
pub struct NodeMem {
pub node: u32,
pub anon_bytes: u64,
pub file_bytes: u64,
pub huge_bytes: u64,
pub shmem_bytes: u64,
}
fn parse_node_meminfo(content: &str) -> Vec<NodeMem> {
// The kernel meminfo format is:
// Node 0 Anon: 12345678 kB
// Node 0 File: 2345678 kB
// ...
// We accumulate fields per node into a HashMap.
use std::collections::HashMap;
let mut node_data: HashMap<u32, NodeMem> = HashMap::new();
for line in content.lines() {
// Lines look like: "Node 0 Anon: 12345678 kB"
let parts: Vec<&str> = line.split_whitespace().collect();
if parts.len() < 4 {
continue;
}
// parts[0] = "Node", parts[1] = node_id, parts[2] = "Anon:", parts[3] = value
if parts[0] != "Node" {
continue;
}
let node_id: u32 = parts[1].parse().unwrap_or(0);
let field_name = parts[2].trim_end_matches(':');
// Values are in KiB; convert to bytes. KiB→bytes is portable
// (1 KiB = 1024 bytes regardless of page size). The old version
// converted to "pages" by dividing by 4, but that assumes 4 KiB
// pages — wrong on ARM64 with 16K or 64K pages. Bytes are unambiguous.
let value_kib: u64 = parts[3].parse().unwrap_or(0);
let value_bytes = value_kib * 1024;
let entry = node_data.entry(node_id).or_insert_with(|| NodeMem {
node: node_id,
anon_bytes: 0,
file_bytes: 0,
huge_bytes: 0,
shmem_bytes: 0,
});
match field_name {
"Anon" => entry.anon_bytes = value_bytes,
"File" => entry.file_bytes = value_bytes,
"HugePages" => entry.huge_bytes = value_bytes,
"Shmem" => entry.shmem_bytes = value_bytes,
_ => {}
}
}
let mut nodes: Vec<NodeMem> = node_data.into_values().collect();
nodes.sort_by_key(|n| n.node);
nodes
}
/// Read per-node memory info from sysfs (all nodes)
fn read_all_node_meminfo() -> std::io::Result<Vec<NodeMem>> {
let node_dir = std::path::Path::new("/sys/devices/system/node");
let mut all_nodes = Vec::new();
for entry in std::fs::read_dir(node_dir)?.flatten() {
let name = entry.file_name();
let name_str = name.to_string_lossy();
if !name_str.starts_with("node") {
continue;
}
let meminfo_path = entry.path().join("meminfo");
if !meminfo_path.exists() {
continue;
}
let content = std::fs::read_to_string(&meminfo_path)?;
let mut parsed = parse_node_meminfo(&content);
all_nodes.append(&mut parsed);
}
all_nodes.sort_by_key(|n| n.node);
Ok(all_nodes)
}
}Storing values as bytes avoids the page-size portability trap. On x86, pages are 4 KiB. On ARM64, pages can be 4K, 16K, or 64K depending on the kernel configuration. The KiB→pages conversion (value_kib / 4) only works on 4K systems. The KiB→bytes conversion (value_kib * 1024) works everywhere — 1 KiB is always 1024 bytes regardless of page size.
/sys/devices/system/node/ — cross-node stats
Linux exposes per-node information through sysfs. The directory structure:
/sys/devices/system/node/
├── node0/
│ ├── cpulist # CPUs on this node
│ ├── distance # NUMA distances to other nodes
│ ├── meminfo # memory stats for this node
│ └── numastat # NUMA hit/miss for this node
├── node1/
│ └── ...
numastat is particularly useful:
numa_hit 12345678
numa_miss 234567
numa_foreign 12345
interleave_hit 1234
other_node 5678
This is per-node data, which is more useful than the system-wide vmstat when you’re debugging a specific NUMA imbalance.
#![allow(unused)]
fn main() {
use std::fs;
fn read_node_numastat(node: u32) -> std::io::Result<NumaStats> {
let path = format!("/sys/devices/system/node/node{}/numastat", node);
let content = fs::read_to_string(&path)?;
let mut stats = NumaStats::default();
for line in content.lines() {
let mut parts = line.split_whitespace();
let name = parts.next().unwrap_or("");
let value: u64 = parts.next().unwrap_or("0").parse().unwrap_or(0);
match name {
"numa_hit" => stats.numa_hit = value,
"numa_miss" => stats.numa_miss = value,
"numa_foreign" => stats.numa_foreign = value,
_ => {}
}
}
// Note: pgmigrate_success and pgmigrate_fail are left at 0 (the
// Default). These fields come from /proc/vmstat, not from per-node
// numastat. If you need per-node migration rates, read vmstat instead.
Ok(stats)
}
}Hugepage utilization
Hugepages (typically 2 MiB or 1 GiB) reduce TLB pressure because one TLB entry covers a much larger region. A regular 4 KiB page needs one TLB entry per 4 KiB of contiguous memory. A 2 MiB hugepage needs one TLB entry per 2 MiB — 512x more coverage per entry. For workloads that scan large arrays or walk page tables, this can halve TLB miss rates.
The data comes from /proc/meminfo:
HugePages_Total: 1024
HugePages_Free: 512
HugePages_Rsvd: 128
HugePages_Surp: 0
AnonHugePages: 2048
The useful metrics:
- Hugepage pool utilization:
(HugePages_Total - HugePages_Free + HugePages_Rsvd) / HugePages_Total. The+ HugePages_Rsvdmatters —HugePages_Freeincludes reserved pages (claimed but not yet faulted in), so the truly free count isHugePages_Free - HugePages_Rsvd. Without accounting for reservations, you’d understate utilization. If utilization is near 100%, the pool is exhausted and new hugepage allocations will fail. If it’s near 0%, the pool is overprovisioned and wasting memory. - Transparent hugepage usage:
AnonHugePages(in KiB) tells you how much transparent hugepage memory is in use. Compare it to the total anonymous memory (AnonPagesin/proc/meminfo) to get a ratio:AnonHugePages / AnonPages. If this ratio is low for a memory-intensive workload, the kernel isn’t coalescing regular pages into hugepages effectively.
Reading it in Rust:
#![allow(unused)]
fn main() {
use std::fs;
#[derive(Default)]
pub struct HugepageStats {
pub total: u64, // HugePages_Total
pub free: u64, // HugePages_Free
pub reserved: u64, // HugePages_Rsvd
pub anon_hugepages: u64, // AnonHugePages (in KiB)
pub anon_pages: u64, // AnonPages (in KiB)
}
fn read_hugepage_stats() -> std::io::Result<HugepageStats> {
let content = fs::read_to_string("/proc/meminfo")?;
let mut stats = HugepageStats::default();
for line in content.lines() {
let mut parts = line.split_whitespace();
let name = parts.next().unwrap_or("");
let value: u64 = parts.next().unwrap_or("0").parse().unwrap_or(0);
match name {
"HugePages_Total:" => stats.total = value,
"HugePages_Free:" => stats.free = value,
"HugePages_Rsvd:" => stats.reserved = value,
"AnonHugePages:" => stats.anon_hugepages = value,
"AnonPages:" => stats.anon_pages = value,
_ => {}
}
}
Ok(stats)
}
fn hugepage_pool_utilization(stats: &HugepageStats) -> f64 {
if stats.total == 0 {
return 0.0;
}
// HugePages_Free includes reserved pages — pages that have been claimed
// by an application but not yet faulted in. A reserved page isn't truly
// free, so the actual free count is (free - reserved). The allocated
// count is total - (free - reserved) = total - free + reserved.
let actual_free = stats.free.saturating_sub(stats.reserved);
(stats.total - actual_free) as f64 / stats.total as f64
}
fn transparent_hugepage_ratio(stats: &HugepageStats) -> f64 {
if stats.anon_pages == 0 {
return 0.0;
}
stats.anon_hugepages as f64 / stats.anon_pages as f64
}
}A pool utilization above 90% means you should increase HugePages_Total in the kernel boot parameters. A transparent hugepage ratio below 10% for a memory-intensive workload means the kernel’s khugepaged daemon isn’t coalescing pages fast enough — check /sys/kernel/mm/transparent_hugepage/ for the current policy (always, madvise, or never).
Next: Part 8 — Uncore IMC Bandwidth — Measure memory bandwidth through the Integrated Memory Controller.
Part 8 — Uncore IMC Bandwidth
The CPU cores get all the attention. The memory controller sits quietly in the corner of the same chip, and nobody thinks about it until memory bandwidth is saturated and everything stalls.
Modern server CPUs put the Integrated Memory Controller (IMC) on the same package as the cores, but outside the cores themselves. Intel calls this the uncore — the silicon on the chip that isn’t a CPU core. It includes the memory controller, the last-level cache, and the mesh interconnect that ties everything together. The uncore has its own performance counters, and they measure memory bandwidth: how many bytes per second are flowing through the memory controller to DRAM.
If memory bandwidth is saturated, adding more CPU cores won’t help. The cores will stall waiting for memory. Monitoring IMC bandwidth tells you whether you’re approaching that ceiling.
What “uncore” means
On Intel, the chip is divided into two parts:
- Core: the CPU cores — instruction execution, cache, registers
- Uncore: everything else — the memory controller, the last-level cache (L3), the mesh interconnect that links the cores to the uncore
The uncore is shared across all cores on the socket. Its counters are accessible via perf_event_open but they live in a separate PMU (Performance Monitoring Unit) from the core PMU. You open them differently.
The IMC counters on Intel
Intel’s uncore IMC events are in the uncore_imc PMU. The type value is architecture-specific:
#![allow(unused)]
fn main() {
// For Intel Skylake and later server CPUs
// The type value is architecture-specific and varies between kernels.
// Don't hardcode it — read it from sysfs (see find_uncore_imc_type below).
const UNCORE_IMC_TYPE_EXAMPLE: u32 = 15; // example only; always read from sysfs
}The reliable way to find the type is through sysfs:
ls /sys/bus/event_source/devices/
Look for something like uncore_imc_0 or uncore_cha_0 (Cache Housing Agent — also useful). The IMC is uncore_imc_<socket_id>.
cat /sys/bus/event_source/devices/uncore_imc_0/type
This prints the integer type value you need for perf_event_open.
The key IMC events (from the Intel perfmon repo, iMC unit):
- CAS count (all): event
0x04, umask0x0F— all CAS (Column Address Strobe) operations, the memory commands that transfer data. This is the most useful bandwidth proxy. - CAS read: event
0x04, umask0x03— DRAM read CAS commands (includes underfill reads) - CAS write: event
0x04, umask0x0C— DRAM write CAS commands - DRAM activate count: event
0x01, umask0x02— DRAM ACT (activate) commands for writes; umask0x01for reads. Cycles the DRAM rank was active.
When you open a raw uncore event with perf_event_open, the config field encodes both the event and umask: config = (umask << 8) | event. So CAS count all (event 0x04, umask 0x0F) becomes config = 0x0F04.
The CAS (Column Address Strobe) counter is the standard way to compute memory bandwidth. Each CAS operation transfers 64 bytes (one cache line). So:
bandwidth_bytes_per_sec = CAS_count * 64 / elapsed_seconds
bandwidth_gb_per_sec = CAS_count * 64 / 1_000_000_000 / elapsed_seconds
Opening uncore IMC counters
Uncore counters have restrictions. They can only be opened:
- With
cpu=-1for system-wide (all sockets) — or with a per-socket fd - By root or a process with
CAP_SYS_ADMIN - With the specific CPU(s) that own the uncore (socket-local CPUs only)
On a multi-socket system, you open one fd per socket:
#![allow(unused)]
fn main() {
use std::fs;
fn find_uncore_imc_type() -> std::io::Result<u32> {
// Find the uncore_imc PMU type value from sysfs.
// On multi-socket systems, each socket has its own uncore_imc_N device
// (uncore_imc_0, uncore_imc_1, etc.), but they all share the same PMU type.
// We read the first one we find.
let entries = std::fs::read_dir("/sys/bus/event_source/devices/")?;
for entry in entries.flatten() {
let name = entry.file_name().into_string().unwrap_or_default();
if name.starts_with("uncore_imc") {
let type_path = entry.path().join("type");
let type_str = fs::read_to_string(&type_path)?.trim().to_owned();
return type_str.parse::<u32>()
.map_err(|e| std::io::Error::new(std::io::ErrorKind::InvalidData, e));
}
}
Err(std::io::Error::new(
std::io::ErrorKind::NotFound,
"uncore_imc PMU not found",
))
}
fn open_imc_counter(
pmu_type: u32,
config: u64,
cpu: libc::c_int,
) -> std::io::Result<libc::c_int> {
// Hand-rolled perf_event_attr — libc::perf_event_attr isn't available
// on all platforms. The field ordering must match the kernel struct exactly.
#[repr(C)]
struct PerfEventAttr {
type_: u32,
size: u32,
config: u64,
sample_period: u64,
sample_type: u64,
read_format: u64,
flags: u64, // bit 0: disabled, bit 2: pinned
}
let attr = PerfEventAttr {
type_: pmu_type,
size: std::mem::size_of::<PerfEventAttr>() as u32,
config,
sample_period: 0,
sample_type: 0,
read_format: 0,
flags: 0b101, // disabled=1 (bit 0), pinned=1 (bit 2)
};
let fd = unsafe {
// pid=-1: uncore PMUs are system-wide, they don't monitor a specific
// process. The kernel requires pid=-1 for uncore events.
libc::syscall(libc::SYS_perf_event_open, &attr as *const _, -1, cpu, -1, 0)
};
if fd < 0 {
Err(std::io::Error::last_os_error())
} else {
Ok(fd as libc::c_int)
}
}
}Computing memory bandwidth
#![allow(unused)]
fn main() {
fn compute_bandwidth_gbps(cas_count_delta: u64, elapsed_secs: f64) -> f64 {
if elapsed_secs <= 0.0 {
return 0.0;
}
// 64 bytes per CAS operation (one cache line)
(cas_count_delta as f64 * 64.0) / elapsed_secs / 1e9
}
}Each CAS operation transfers one 64-byte cache line, so the math is straightforward.
The enable_counter and read_counter helper functions are the same ones from Part 3 — they call ioctl(fd, PERF_EVENT_IOC_ENABLE, 0) and read(fd, &mut value) respectively.
Counter width and wrap-around
Intel uncore IMC counters are 48-bit on Xeon Scalable processors (Skylake through Emerald Rapids). A 48-bit counter overflows after 2^48 ≈ 281 trillion events. At peak DDR5 bandwidth (~400 GB/s, or about 6.25 billion CAS operations per second), that’s roughly 12.5 hours before the counter wraps.
If your monitoring daemon polls every 1–10 seconds, you won’t see a wrap — the delta between reads is small. But if you open a counter, go to sleep for 13 hours, and then read it, the value will have wrapped and your delta computation (curr - prev) produces a huge number. The saturating_sub in the polling loop prevents overflow in Rust, but the resulting bandwidth value will be wrong.
For long-running monitors, poll frequently enough that the counter can’t wrap between reads (every few seconds is more than sufficient). If you need to handle arbitrary gaps, check whether curr < prev — that indicates a wrap, and you need to account for the full counter range: delta = (2^48 - prev) + curr.
The full polling loop
#![allow(unused)]
fn main() {
use std::time::Instant;
struct ImcBandwidth {
pub bandwidth_gbps: f64,
pub cas_count: u64,
pub elapsed_secs: f64,
}
fn open_socket_imc_counters(
pmu_type: u32,
socket_cpus: &[i32], // CPUs local to each socket
) -> std::io::Result<Vec<(i32, i32)>> {
// Open one CAS counter fd per socket. Keep these fds open for the
// lifetime of the monitor — don't open/close on every poll.
//
// socket_cpus: one CPU id per socket (any CPU local to that socket's
// uncore works). On a single-socket system, this is [0]. On dual-socket,
// find them with:
// cat /sys/devices/system/node/node0/cpulist → socket 0 CPUs
// cat /sys/devices/system/node/node1/cpulist → socket 1 CPUs
let mut fds = Vec::new();
for (socket, &cpu) in socket_cpus.iter().enumerate() {
let fd = open_imc_counter(pmu_type, 0x0F04, cpu)?; // CAS count all (event 0x04, umask 0x0F)
enable_counter(fd)?;
fds.push((socket as i32, fd));
}
Ok(fds)
}
fn read_imc_counts(socket_fds: &[(i32, i32)]) -> std::io::Result<Vec<(i32, u64)>> {
let mut results = Vec::new();
for &(socket, fd) in socket_fds {
let cas = read_counter(fd)?;
results.push((socket, cas));
}
Ok(results)
}
fn poll_imc(
socket_fds: &[(i32, i32)],
prev_counts: &[(i32, u64)],
interval_secs: f64,
) -> std::io::Result<Vec<ImcBandwidth>> {
let current_counts = read_imc_counts(socket_fds)?;
let mut result = Vec::new();
for ((socket, prev_cas), (_, curr_cas)) in prev_counts.iter().zip(¤t_counts) {
let delta = curr_cas.saturating_sub(*prev_cas);
result.push(ImcBandwidth {
bandwidth_gbps: compute_bandwidth_gbps(delta, interval_secs),
cas_count: delta,
elapsed_secs: interval_secs,
});
}
Ok(result)
}
}Virtualization caveat
You can’t read uncore counters inside a virtual machine (in most cases). The guest doesn’t have direct access to the IMC hardware. When you try to open an uncore event from inside a guest, you get EPERM or EINVAL.
If you’re building a monitoring agent that runs inside VMs, skip uncore IMC reading — it’s not available. For bare-metal hosts, it’s one of the most useful performance signals.
AMD alternative: Data Fabric counters
AMD’s equivalent of the Intel IMC counters are the Data Fabric (DF) performance counters. The Data Fabric is the interconnect that links AMD CPU cores, memory controllers, and I/O — analogous to Intel’s mesh + uncore. DF counters track memory bandwidth through the same perf_event_open interface, but the PMU names and event encodings are different.
Look for uncore_data_fabric or amd_df PMUs in sysfs:
ls /sys/bus/event_source/devices/
# Look for uncore_data_fabric or amd_df on AMD systems
AMD also has Instruction-Based Sampling (IBS) — a different approach that periodically samples the instruction stream and reports details about memory operations. IBS is more useful for profiling (“where are the loads that miss in L3?”) than for bandwidth monitoring. If you need per-instruction latency data, IBS is the tool. If you need aggregate bandwidth, use the DF counters.
IBS events are in the ibs_op and ibs_fetch PMUs on AMD. These are not available on Intel.
Next: Part 9 — Thermal Monitoring — Read thermal zones from sysfs and compute headroom before throttling.
Part 9 — Thermal Monitoring
A CPU running hot doesn’t jump straight to throttling — there are graduated stages, and each one leaves a signal you can read.
Thermal throttling is the last resort. Before the CPU hits its critical temperature and starts skipping cycles, the kernel has already entered passive cooling — reducing the clock speed to lower heat output. If you’re watching CPU utilization stay flat while your benchmark scores drop, passive cooling is probably the cause. The good news: you can see it happening in real time.
Linux exposes thermal zone readings through sysfs. Each thermal zone corresponds to a physical temperature sensor somewhere in the system.
What thermal zones are
The kernel abstracts temperature sensors as thermal zones. A thermal zone has a type (what it measures), a temp (current temperature in milli-degrees Celsius), and a set of trip_point_* thresholds.
Common zone types on x86:
x86_pkg_temp— the CPU package (whole-chip temperature)acpitz— ACPI thermal zone (usually near the CPU)coretemp— per-core temperature (Intel)nvme— NVMe drive temperaturetztsx— various other sensors
On ARM servers:
soc-thermal— the SoC temperaturecpu-thermal— the CPU cluster temperature
The zone naming and quantity vary by hardware. The kernel creates whatever zones the hardware exposes.
Reading thermal zones from sysfs
The sysfs layout:
/sys/class/thermal/
├── thermal_zone0/
│ ├── type # "x86_pkg_temp", "acpitz", etc.
│ ├── temp # current temperature in millidegrees Celsius (e.g., 72000 = 72.0°C)
│ ├── trip_point_0_temp # threshold in m°C
│ ├── trip_point_0_type # "active", "passive", "hot", "critical"
│ └── ...
├── thermal_zone1/
│ └── ...
Each zone has multiple trip points — temperature thresholds that trigger different cooling responses. The types:
- passive: fan speeds up (no performance impact)
- active: stronger cooling, minor performance impact
- hot: thermal throttle imminent
- critical: emergency shut down if reached
Parsing a thermal zone
#![allow(unused)]
fn main() {
use std::fs;
#[derive(Debug, Clone)]
pub struct ThermalZone {
pub name: String, // zone name in sysfs (e.g., "thermal_zone0")
pub zone_type: String, // what this zone measures (e.g., "x86_pkg_temp")
pub temp_millicelsius: i64,
pub trip_points: Vec<TripPoint>,
}
#[derive(Debug, Clone)]
pub struct TripPoint {
pub temp_millicelsius: i64,
pub trip_type: String, // "passive", "active", "hot", "critical"
}
fn read_thermal_zone(zone_path: &std::path::Path) -> std::io::Result<ThermalZone> {
let name = zone_path
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("unknown")
.to_string();
let zone_type = fs::read_to_string(zone_path.join("type"))?.trim().to_string();
let temp_str = fs::read_to_string(zone_path.join("temp"))?.trim().to_string();
let temp_millicelsius: i64 = temp_str.parse().unwrap_or(0);
let mut trip_points = Vec::new();
// Trip points are numbered starting at 0
let mut idx = 0;
loop {
let trip_type_path = zone_path.join(format!("trip_point_{}_type", idx));
let trip_temp_path = zone_path.join(format!("trip_point_{}_temp", idx));
if !trip_type_path.exists() {
break;
}
let trip_type = fs::read_to_string(&trip_type_path)?.trim().to_string();
let trip_temp_str = fs::read_to_string(&trip_temp_path)?.trim().to_string();
let trip_temp: i64 = trip_temp_str.parse().unwrap_or(0);
trip_points.push(TripPoint {
temp_millicelsius: trip_temp,
trip_type,
});
idx += 1;
}
Ok(ThermalZone {
name,
zone_type,
temp_millicelsius,
trip_points,
})
}
fn read_all_thermal_zones() -> std::io::Result<Vec<ThermalZone>> {
let thermal_path = std::path::Path::new("/sys/class/thermal");
let entries = fs::read_dir(thermal_path)?;
let mut zones = Vec::new();
for entry in entries.flatten() {
let path = entry.path();
if path.file_name().and_then(|n| n.to_str())
.map(|n| n.starts_with("thermal_zone"))
.unwrap_or(false)
{
if let Ok(zone) = read_thermal_zone(&path) {
zones.push(zone);
}
}
}
Ok(zones)
}
}Computing thermal headroom
Thermal headroom is the gap between the current temperature and the critical threshold:
#![allow(unused)]
fn main() {
fn thermal_headroom(zone: &ThermalZone) -> Option<i64> {
// If there are multiple critical trip points, use the lowest one —
// that's the threshold the CPU will hit first.
let critical = zone.trip_points.iter()
.filter(|tp| tp.trip_type == "critical")
.map(|tp| tp.temp_millicelsius)
.min()?;
// headroom = critical - current (both in m°C)
Some(critical - zone.temp_millicelsius)
}
}Headroom is your safety margin. If a workload pushes the CPU toward critical temperature, the headroom shrinks. When headroom hits zero, throttling kicks in.
Thermal headroom in degrees Celsius:
#![allow(unused)]
fn main() {
fn headroom_celsius(zone: &ThermalZone) -> Option<f64> {
thermal_headroom(zone).map(|hm| hm as f64 / 1000.0)
}
}Identifying the package sensor vs. per-core sensors
The most useful zone for CPU performance is the package-level zone. It’s typically x86_pkg_temp (Intel) or the ACPI zone near the CPU. Per-core zones (coretemp) are more granular but package-level is what you watch for overall thermal throttle risk.
#![allow(unused)]
fn main() {
fn find_package_zone(zones: &[ThermalZone]) -> Option<&ThermalZone> {
// x86_pkg_temp is the canonical package-level sensor on Intel
zones.iter()
.find(|z| z.zone_type == "x86_pkg_temp")
.or_else(|| zones.iter().find(|z| z.zone_type.contains("pkg")))
.or_else(|| zones.iter().find(|z| z.zone_type == "acpitz"))
}
}Polling interval
Thermal changes are slow. A CPU at 60°C doesn’t jump to 90°C in a second — the thermal mass is too large. A 1-second polling interval is more than enough. Even 5 seconds is fine for thermal monitoring.
The important thing is to watch for the trend, not individual readings. If the package temperature is creeping up over a 30-second window, something is building heat.
#![allow(unused)]
fn main() {
use std::time::{Duration, Instant};
async fn poll_thermal(interval: Duration) -> anyhow::Result<()> {
loop {
let zones = read_all_thermal_zones()?;
let package = find_package_zone(&zones);
if let Some(pkg) = package {
let current = pkg.temp_millicelsius as f64 / 1000.0;
let headroom = thermal_headroom(pkg).map(|h| h as f64 / 1000.0);
println!(
"package_temp={:.1}°C headroom={:.1}°C type={}",
current,
headroom.unwrap_or(-999.0),
pkg.zone_type,
);
}
tokio::time::sleep(interval).await;
}
}
}Cross-architecture differences
On AMD EPYC, the thermal zones may be named differently. Use ls /sys/class/thermal/ on the target system to see what’s available. The ACPI thermal zones are the most portable fallback — the ACPI spec requires them on all compliant systems.
On ARM servers, the sensor landscape is more fragmented. soc-thermal and cpu-thermal are common names. Some ARM platforms expose only one thermal zone for the whole SoC.
Thresholds at a glance
For quick reference, the thermal throttle scale (Intel desktop/server):
| Temperature | What it means |
|---|---|
| < 70°C | Normal operation, no throttling |
| 70-85°C | Active cooling engaged, performance nominal |
| 85-95°C | Passive cooling — clock speed reduced |
| 95-100°C | Hot — aggressive throttling |
| > 100°C | Critical — emergency throttle |
These thresholds are approximate and vary by SKU. The trip points from sysfs are the authoritative source for your specific hardware.
Next: Part 10 — Block I/O Tracing — Trace block I/O requests and compute IOPS, throughput, and access pattern entropy.
Part 10 — Block I/O Tracing
I/O patterns tell you whether your storage is being used well.
High IOPS with low throughput means many small operations — small random reads, metadata-heavy workloads, or many tiny writes. Low IOPS with high throughput means a few large sequential operations — streaming reads, copy operations. Both are normal, but both can become bottlenecks.
The block layer tracepoints let us observe every I/O request as it enters and completes.
The block tracepoints
Linux has several block I/O tracepoints. The two we care about:
| Tracepoint | When it fires |
|---|---|
block:block_bio_queue | A request is submitted to the block layer |
block:block_bio_complete | A request completes |
The arguments for block:block_bio_queue:
Offset Type Field
------ ---- -----
0 u32 dev (dev_t: encoded device number — decoded below)
4 (4 bytes padding)
8 u64 sector // starting sector number
16 u32 nr_sector // number of sectors
20 char[8] rwbs // R/W/S flag string ("R", "W", etc.)
28 char[16] comm // process name (TASK_COMM_LEN)
The dev field is a 4-byte dev_t followed by 4 bytes of padding to align the 8-byte sector field. The rwbs field is an 8-byte character array that encodes the I/O direction: "R" for read, "W" for write, "S" for sync, "F" for flush, "D" for discard, "N" for none. The comm field is the process name that submitted the I/O.
Important: block_bio_queue is a block_bio-class tracepoint — it operates on struct bio, not struct request. The block_rq-class tracepoints (block_rq_insert, block_rq_issue) have a bytes field and op_flags field that block_bio lacks. If you need the transfer size in bytes, compute it from nr_sector * 512 (each sector is 512 bytes). This is what the eBPF code below does.
Verifying offsets. The offsets above use the
read_atconvention — they start from the first byte after the 8-byte common tracepoint header. The format file includes this header, so its offsets are 8 bytes larger. See the verification note in Part 2 for the full procedure.
The device number dev_t encodes major and minor into a single 32-bit integer. The kernel’s format (since Linux 2.6) packs them as major << 8 | minor for small device numbers, with extended encoding for large majors and minors. The block tracepoints store raw dev_t — you decode it with the standard formulas: major = (dev >> 8) & 0xFFF, minor = (dev & 0xFF) | ((dev >> 12) & 0xFFF00). To get a readable device name, map it through /sys/dev/block/:
#![allow(unused)]
fn main() {
use std::fs;
// Convert dev_t to a readable device name like "sda" or "nvme0n1"
fn dev_t_to_name(dev: u32) -> Option<String> {
// Decode dev_t using the kernel's encoding:
// major = (dev >> 8) & 0xFFF
// minor = (dev & 0xFF) | ((dev >> 12) & 0xFFF00)
// This matches the MAJOR(dev)/MINOR(dev) macros from <sys/sysmacros.h>.
// The block tracepoints store raw dev_t, not new_encode_dev().
let major = (dev >> 8) & 0xFFF;
let minor = (dev & 0xFF) | ((dev >> 12) & 0xFFF00);
let path = format!("/sys/dev/block/{}:{}", major, minor);
// read_link returns a relative path like "../../devices/.../block/sda"
// The last component after "/block/" is the device name.
let target = fs::read_link(&path).ok()?;
let target_str = target.into_os_string().into_string().ok()?;
// Extract the device name: everything after the last "/block/" component
if let Some(idx) = target_str.rfind("/block/") {
Some(target_str[idx + 7..].to_owned())
} else {
// Fallback: use the last path component
target_str.split('/').next_back().map(|s| s.to_owned())
}
}
}The /sys/dev/block/MAJOR:MINOR symlink points into the device tree — something like ../../devices/pci0000:00/.../block/sda. We extract the device name (sda, nvme0n1) from the block/ component rather than returning the raw symlink target, which would be a long relative path.
For our metrics, we’ll track per-device IOPS and throughput.
The eBPF program
#![allow(unused)]
fn main() {
// ebpf-programs/src/blockio.rs
use aya_ebpf::programs::TracePointContext;
use aya_ebpf::macros::tracepoint;
use aya_ebpf::maps::{HashMap, PerCpuArray, RingBuf};
use aya_ebpf::helpers::bpf_ktime_get_ns;
#[derive(Clone, Copy)]
#[repr(C)]
pub struct BioQueueEvent {
pub dev: u32,
pub sector: u64,
pub nr_sector: u32,
pub timestamp: u64,
}
// Per-device counter: dev → (ops_count, total_bytes)
// HashMap is shared across CPUs, so concurrent updates from different CPUs
// can lose counts (read-modify-write race). For monitoring where approximate
// counts are acceptable, this is fine. For exact counts, use PerCpuArray
// and sum in userspace (like the histogram in Part 12).
#[map]
static IO_COUNTERS: HashMap<u32, (u64, u64)> = HashMap::with_max_entries(64, 0);
// Per-CPU sampling counter — avoids static mut by using a map
#[map]
static SAMPLE_COUNTER: PerCpuArray<u64> = PerCpuArray::with_max_entries(1, 0);
// Ring buffer for sector samples (sent every 100th event)
#[map]
static SECTOR_SAMPLES: RingBuf = RingBuf::with_byte_size(8 * 4096, 0);
#[derive(Clone, Copy)]
#[repr(C)]
pub struct SectorSample {
pub dev: u32,
pub sector: u64,
}
#[tracepoint]
pub fn block_bio_queue(ctx: TracePointContext) -> u32 {
let dev = unsafe { ctx.read_at::<u32>(0).unwrap_or(0) };
let sector = unsafe { ctx.read_at::<u64>(8).unwrap_or(0) };
let nr_sector = unsafe { ctx.read_at::<u32>(16).unwrap_or(0) };
// Update per-device counters (per-CPU copy via HashMap keyed by dev)
unsafe {
let (ops, bytes) = IO_COUNTERS.get(&dev).copied().unwrap_or((0u64, 0u64));
let new_ops = ops + 1;
let new_bytes = bytes + (nr_sector as u64 * 512);
let _ = IO_COUNTERS.insert(&dev, &(new_ops, new_bytes), 0);
}
// Sampling: every 100th operation per CPU — avoid static mut
unsafe {
if let Some(ptr) = SAMPLE_COUNTER.get_ptr_mut(0) {
*ptr += 1;
if *ptr % 100 == 0 {
let sample = SectorSample { dev, sector };
SECTOR_SAMPLES.output(&sample, 0);
}
}
}
0
}
}A few things to notice:
Tuple values in HashMap: HashMap<u32, (u64, u64)> stores a tuple as the value. This works in Aya eBPF — the value size is the size of the tuple. It’s a convenient way to store multiple counters per key.
PerCpuArray<u64> for the sampling counter: The original code used static mut COUNTER: u64 which is forbidden in safe Rust and problematic in eBPF. Using a PerCpuArray with one entry gives each CPU its own counter — no atomics needed.
unsafe on get(): HashMap::get() is unsafe because the kernel doesn’t guarantee atomicity without BPF_F_NO_PREALLOC. For metrics aggregation, occasional lost updates are acceptable.
Computing IOPS and throughput
In userspace, read the counters and compute rates:
#![allow(unused)]
fn main() {
use aya::maps::HashMap as AyaHashMap;
pub struct IoStats {
pub device: String,
pub iops: f64,
pub throughput_mbps: f64,
pub ops_count: u64,
pub bytes_count: u64,
}
fn read_io_stats(
counters: &AyaHashMap<u32, (u64, u64)>,
prev: &std::collections::HashMap<u32, (u64, u64)>,
elapsed_secs: f64,
) -> Vec<IoStats> {
let safe_elapsed = if elapsed_secs > 0.0 { elapsed_secs } else { 1.0 };
counters.iter().filter_map(|item| {
let (dev, (ops, bytes)) = item.ok()?;
let prev_data = prev.get(&dev).copied().unwrap_or((0, 0));
let ops_delta = ops.saturating_sub(prev_data.0);
let bytes_delta = bytes.saturating_sub(prev_data.1);
Some(IoStats {
device: dev_t_to_name(dev).unwrap_or_else(|| format!("{:08x}", dev)),
iops: ops_delta as f64 / safe_elapsed,
throughput_mbps: (bytes_delta as f64 / safe_elapsed) / 1e6,
ops_count: ops_delta,
bytes_count: bytes_delta,
})
}).collect()
}
}IOPS entropy — measuring randomness
High IOPS can mean two different things:
- Sequential I/O: reading a large file in order, one big read per access — predictable, batchable
- Random I/O: reading many small blocks at scattered addresses — unpredictable, hard to batch
Both have high IOPS. The difference is in the sector number distribution. Shannon entropy of the sector numbers measures how predictable or random the access pattern is.
Here’s how to compute it:
#![allow(unused)]
fn main() {
fn compute_entropy(sectors: &[u64], num_buckets: usize) -> f64 {
if sectors.is_empty() {
return 0.0;
}
// Bucket sectors into ranges to build a histogram
let min = *sectors.iter().min().unwrap_or(&0);
let max = *sectors.iter().max().unwrap_or(&0);
if min == max {
return 0.0; // all accesses in one bucket = completely predictable
}
let range = max - min + 1;
let bucket_size = (range / num_buckets as u64).max(1);
let mut counts = vec![0usize; num_buckets];
for §or in sectors {
let bucket = ((sector - min) / bucket_size) as usize;
let bucket = bucket.min(num_buckets - 1);
counts[bucket] += 1;
}
let total = counts.iter().sum::<usize>() as f64;
if total == 0.0 {
return 0.0;
}
// H = -sum(p * log2(p)) for each bucket
let mut entropy = 0.0;
for &count in &counts {
if count == 0 {
continue;
}
let p = count as f64 / total;
entropy -= p * p.log2();
}
entropy
}
}Entropy is measured in bits. A value near 0 means the accesses are concentrated in one bucket — very sequential. A value near log2(num_buckets) means accesses are spread evenly across all buckets — very random.
For num_buckets = 16:
- 0-2 bits: sequential (always reading from one area)
- 2-3 bits: some locality (reading from a few areas)
- 3-4 bits: random (reading from across the address space)
Ring buffer reader for sector sampling
For entropy calculation, you need the actual sector numbers, not just counts. The eBPF program sends every 100th sector number to the ring buffer:
#![allow(unused)]
fn main() {
use aya::maps::RingBuf;
use aya::Ebpf;
#[derive(Clone, Copy, Debug)]
#[repr(C)]
pub struct SectorSample {
pub dev: u32,
pub sector: u64,
}
async fn poll_sector_samples(
ring_buf: &mut aya::maps::RingBuf,
sector_window: &mut VecDeque<u64>,
) -> anyhow::Result<()> {
while let Some(item) = ring_buf.next() {
let sample = unsafe {
std::ptr::read_unaligned(item.as_ptr() as *const SectorSample)
};
sector_window.push_back(sample.sector);
}
// Keep the last 10000 samples
while sector_window.len() > 10000 {
sector_window.pop_front();
}
Ok(())
}
}The ring buffer stores raw bytes. item is &[u8] — you cast it back to the struct type with read_unaligned. This is the standard pattern for receiving structured events from the ring buffer.
Userspace aggregation
#![allow(unused)]
fn main() {
use std::collections::{HashMap, VecDeque};
use std::time::Duration;
async fn poll_block_io(
counters: &mut aya::maps::HashMap<u32, (u64, u64)>,
ring_buf: &mut aya::maps::RingBuf,
prev: &mut std::collections::HashMap<u32, (u64, u64)>,
sector_window: &mut VecDeque<u64>,
elapsed_secs: f64,
) -> anyhow::Result<()> {
// Drain ring buffer samples for entropy calculation
poll_sector_samples(ring_buf, sector_window).await?;
// Read the eBPF counters map
// (counters map is passed in, already opened once at startup)
let safe_elapsed = if elapsed_secs > 0.0 { elapsed_secs } else { 1.0 };
// Compute IOPS and throughput by iterating the eBPF map and comparing to prev
let mut stats = Vec::new();
for item in counters.iter() {
let (dev, (ops, bytes)) = item?;
let (prev_ops, prev_bytes) = prev.get(&dev).copied().unwrap_or((0, 0));
let ops_delta = ops.saturating_sub(prev_ops);
let bytes_delta = bytes.saturating_sub(prev_bytes);
stats.push(IoStats {
device: dev_t_to_name(dev).unwrap_or_else(|| format!("{:08x}", dev)),
iops: ops_delta as f64 / safe_elapsed,
throughput_mbps: (bytes_delta as f64 / safe_elapsed) / 1e6,
ops_count: ops_delta,
bytes_count: bytes_delta,
});
prev.insert(dev, (ops, bytes));
}
let entropy = compute_entropy(§or_window.iter().copied().collect::<Vec<_>>(), 16);
for stat in stats {
println!(
"{}: iops={:.0} mbps={:.1} entropy={:.2}",
stat.device, stat.iops, stat.throughput_mbps, entropy
);
}
Ok(())
}
}Next: Part 11 — vhost and Virtio Ring Instrumentation — Instrument the virtio ring with kprobes to measure I/O latency at the virtualization boundary.
Part 11 — vhost and Virtio Ring Instrumentation
If you run virtual machines on Linux, the guest and the host need to move data back and forth for disk I/O and networking. That data path has a bottleneck: the shared memory ring where work items are posted. When that ring fills up, the guest waits. When the host can’t process descriptors fast enough, the guest waits longer.
The ring is called the virtqueue — a circular buffer in shared memory. The guest writes descriptors (“here’s a network packet to send” or “read 4KB from this disk offset”) into the ring. The host-side vhost kernel module picks them up, does the work, and signals completion. If you can measure how full the ring is and how long descriptors sit before being processed, you can see I/O latency building up before it shows up in the guest’s metrics.
Understanding the ring means understanding the latency and throughput of your VM’s I/O path.
Virtio ring basics
The virtio ring has three parts:
- Descriptor table: an array of data buffer descriptors — each entry points to a buffer and its length
- Available ring: an array of descriptor indices the guest has made available to the host
- Used ring: an array of descriptor indices the host has processed and returned to the guest
The host consumer reads from the available ring and writes completions to the used ring. The guest producer writes to the available ring and reads from the used ring.
The key performance questions:
- How many descriptors are being added per second?
- Is the ring getting full (stall condition)?
- What’s the latency from add to completion?
KProbes vs tracepoints
Tracepoints are stable hooks placed by kernel developers. KProbes are dynamic — you can attach to any kernel function. They’re more powerful but less portable.
vhost and virtio don’t have tracepoints for the fast path operations. To instrument the ring, we need kprobes.
The tradeoff: kprobe function names change between kernel versions. A probe that works on kernel 5.15 might not exist on 6.1. This is the cost of deep kernel instrumentation.
Finding the right symbols
The kernel exposes all exported symbols in /proc/kallsyms. Look for vhost and virtqueue functions:
# Find vhost functions
cat /proc/kallsyms | grep -i vhost | grep -v "\[.*\]" | head -20
# Find virtqueue functions
cat /proc/kallsyms | grep -i virtqueue | grep -v "\[.*\]" | head -20
Common probe targets:
| Function | What it does |
|---|---|
handle_tx_kick | TX kick from the guest (vhost-net) — actually exported |
handle_rx_kick | RX kick from the guest (vhost-net) — actually exported |
vhost_scsi_handle_vq | SCSI I/O submission (vhost-scsi) — actually exported |
vhost_worker | The main vhost worker loop — actually exported |
⚠️ Why not __virtqueue_add? __virtqueue_add and virtqueue_add are declared static inline in the kernel source (drivers/virtio/virtio_ring.c). The compiler expands them at the call site, so they don’t appear in /proc/kallsyms and can’t be probed with kprobes. Any guide that shows __virtqueue_add as a kprobe target won’t work on a standard kernel build. The functions in the table above are exported and appear in kallsyms.
The exact names vary by kernel version and configuration. Build discovery into your program.
A note on struct access
KProbes give you access to function arguments, not named struct fields. You access arguments through ctx.arg::<T>(n) — the nth argument (0-indexed). To read struct fields from pointers, use bpf_probe_read.
Tracepoints decode struct fields for you — the kernel publishes the layout in /sys/kernel/tracing/events/.../format, and you read named fields at known offsets. KProbes don’t get that luxury. You get raw function arguments, and if one of those arguments is a pointer to a struct, you need to read the struct fields yourself with bpf_probe_read.
For production use, there are two approaches:
- BTF (Compile Once, Run Everywhere): access struct fields by name, with offsets resolved at load time from the kernel’s BTF data
- Hardcoded offsets: read struct fields at fixed offsets — works but breaks when the kernel changes the struct layout
For this tutorial, we’ll use the argument-based approach (reading function arguments directly), which works without knowing any struct layouts.
Counting descriptor additions
The most fundamental metric: how many descriptors are being processed per second per queue. We can’t easily get the queue index from a kprobe without CO-RE and BTF, so we’ll count per-CPU as a proxy:
#![allow(unused)]
fn main() {
// ebpf-programs/src/vhost.rs
use aya_ebpf::programs::ProbeContext;
use aya_ebpf::macros::{kprobe, map};
use aya_ebpf::maps::{HashMap, RingBuf};
use aya_ebpf::helpers::{bpf_ktime_get_ns, bpf_get_smp_processor_id};
#[map]
static VQ_COUNTER: HashMap<u32, u64> = HashMap::with_max_entries(256, 0);
// The #[kprobe] macro marks this function as a kprobe entry point.
// The kernel symbol to probe is specified at attach time from userspace.
#[kprobe]
pub fn vhost_kick(ctx: ProbeContext) -> u32 {
// We're probing handle_tx_kick or handle_rx_kick — actually exported functions
// that process virtqueue entries. We can't easily get the queue index
// without CO-RE. For now, count per CPU as a proxy.
let cpu = unsafe { bpf_get_smp_processor_id() };
unsafe {
let count = VQ_COUNTER.get(&cpu).copied().unwrap_or(0u64);
let new_count = count + 1;
let _ = VQ_COUNTER.insert(&cpu, &new_count, 0);
}
0
}
}ctx.arg::<T>(n) returns Option<T>. When reading a pointer argument with ctx.arg::<*const T>(n), the result is Option<*const T>. None means the read itself failed (the argument index is out of bounds, or the BPF verifier rejected the read). A null pointer value comes back as Some(std::ptr::null()), not None — check for null explicitly if the function might pass one.
Getting queue index with CO-RE
If your kernel has BTF enabled (most modern kernels do), you can access struct fields by name. This lets you get the queue index from the vhost_virtqueue struct and track per-queue counters instead of per-CPU.
CO-RE (Compile Once, Run Everywhere) is Aya’s mechanism for portable BTF-based access. It works by reading struct field offsets at load time using BTF data from the target kernel. With CO-RE, you’d read the handle_kick argument to get the vhost_virtqueue pointer, then use bpf_core_read to read the index field by name.
A complete CO-RE example is beyond the scope of this tutorial — it requires kernel BTF data to be available (/sys/kernel/btf/vmlinux must exist), Aya’s aya-obj BTF parsing, and careful struct definitions that match the kernel’s layout. The Aya project has CO-RE examples in their repository that show the full pattern.
If BTF isn’t available on your target kernel, fall back to the per-CPU counting approach shown earlier. Per-CPU counters give you some visibility without the BTF dependency.
Stall detection
A ring stall happens when the available ring is full — the guest has posted descriptors faster than the host can process them, and the ring backs up. When that happens, the next descriptor submission blocks until the host catches up.
We can’t read the ring’s fill level directly without CO-RE (more on that below). Instead, we’ll use a proxy: measure the time between consecutive descriptor processing events on the same CPU. If the gap spikes, something was waiting. But there’s a catch — a long gap can also mean the ring was idle (no I/O submitted). This proxy can’t distinguish a stall from an idle period on its own. In practice, you’d correlate spikes with the vhost worker’s CPU usage or the VM’s I/O queue depth to filter out idle gaps.
#![allow(unused)]
fn main() {
// ebpf-programs/src/vhost.rs
use aya_ebpf::programs::ProbeContext;
use aya_ebpf::macros::{kprobe, map};
use aya_ebpf::maps::{HashMap, RingBuf};
use aya_ebpf::helpers::{bpf_ktime_get_ns, bpf_get_smp_processor_id};
#[map]
static LAST_ADD_TS: HashMap<u32, u64> = HashMap::with_max_entries(256, 0);
#[map]
static STALL_EVENTS: RingBuf = RingBuf::with_byte_size(8 * 4096, 0);
#[derive(Clone, Copy)]
#[repr(C)]
pub struct StallEvent {
pub cpu_id: u32,
pub gap_ns: u64,
}
const STALL_THRESHOLD_NS: u64 = 1_000_000; // 1ms
#[kprobe]
pub fn vhost_kick_stall(ctx: ProbeContext) -> u32 {
let cpu = unsafe { bpf_get_smp_processor_id() };
let now = unsafe { bpf_ktime_get_ns() };
unsafe {
if let Some(&prev_ts) = LAST_ADD_TS.get(&cpu) {
let gap = now.saturating_sub(prev_ts);
if gap > STALL_THRESHOLD_NS {
let event = StallEvent {
cpu_id: cpu,
gap_ns: gap,
};
STALL_EVENTS.output(&event, 0);
}
}
let _ = LAST_ADD_TS.insert(&cpu, &now, 0);
}
0
}
}The full vhost eBPF program
The program below combines the per-CPU descriptor counter and the stall detector into a single function. Both maps (VQ_COUNTER and LAST_ADD_TS / STALL_EVENTS) are updated on every kick event — there’s no need for two separate kprobe functions when both metrics come from the same trigger.
#![allow(unused)]
fn main() {
// ebpf-programs/src/vhost.rs
use aya_ebpf::programs::ProbeContext;
use aya_ebpf::macros::kprobe;
use aya_ebpf::maps::{HashMap, RingBuf};
use aya_ebpf::helpers::{bpf_ktime_get_ns, bpf_get_smp_processor_id};
#[map]
static VQ_COUNTER: HashMap<u32, u64> = HashMap::with_max_entries(256, 0);
#[map]
static LAST_ADD_TS: HashMap<u32, u64> = HashMap::with_max_entries(256, 0);
#[map]
static STALL_EVENTS: RingBuf = RingBuf::with_byte_size(8 * 4096, 0);
#[derive(Clone, Copy)]
#[repr(C)]
pub struct StallEvent {
pub cpu_id: u32,
pub gap_ns: u64,
}
const STALL_THRESHOLD_NS: u64 = 1_000_000;
#[kprobe]
pub fn vhost_kick(ctx: ProbeContext) -> u32 {
let cpu = unsafe { bpf_get_smp_processor_id() };
let now = unsafe { bpf_ktime_get_ns() };
// Per-CPU descriptor counter
unsafe {
let count = VQ_COUNTER.get(&cpu).copied().unwrap_or(0u64);
let new_count = count + 1;
let _ = VQ_COUNTER.insert(&cpu, &new_count, 0);
}
// Stall detection: if the gap since the last kick exceeds
// the threshold, emit a stall event. This is a proxy — a long
// gap might mean a stall (ring full, host backed up) or just
// an idle period (no I/O submitted). Correlate with the
// vhost worker's CPU usage or the VM's I/O queue depth to
// distinguish the two cases.
unsafe {
if let Some(&prev_ts) = LAST_ADD_TS.get(&cpu) {
let gap = now.saturating_sub(prev_ts);
if gap > STALL_THRESHOLD_NS {
let event = StallEvent {
cpu_id: cpu,
gap_ns: gap,
};
STALL_EVENTS.output(&event, 0);
}
}
let _ = LAST_ADD_TS.insert(&cpu, &now, 0);
}
0
}
}Attaching probes dynamically from userspace
The challenge with kprobes: function names change between kernel versions. The solution is to look up the symbol at runtime and attach to whatever exists:
#![allow(unused)]
fn main() {
// monitor/src/vhost.rs
use aya::programs::KProbe;
use std::process::Command;
pub fn attach_vhost_probes(ebpf: &mut aya::Ebpf) -> anyhow::Result<()> {
// The eBPF program is named "vhost_kick" (matching the Rust function).
// At attach time, we specify the kernel symbol to probe.
// Look up available symbols and attach to whichever exists.
//
// Note: __virtqueue_add and virtqueue_add are static inline — they won't
// appear in kallsyms. We probe the exported vhost handler functions instead.
let candidates = [
"handle_tx_kick", // vhost-net TX (most common)
"handle_rx_kick", // vhost-net RX
"vhost_scsi_handle_vq", // vhost-scsi
];
let kallsyms = std::fs::read_to_string("/proc/kallsyms")?;
// Find the eBPF program by its Rust name, not the kernel symbol
let program = ebpf.program_mut("vhost_kick")
.ok_or_else(|| anyhow::anyhow!("vhost_kick program not found in eBPF object"))?;
let kprobe: &mut KProbe = program.try_into()?;
kprobe.load()?;
// Try each candidate kernel symbol. We break after the first
// successful attach because a single KProbe program can only
// be attached to one symbol. If you want to probe multiple
// vhost handlers (e.g., both TX and RX), you need separate
// program instances — one per symbol.
let mut attached = false;
for sym in candidates {
if kallsyms.lines().any(|l| l.contains(sym)) {
kprobe.attach(sym, 0)?; // 0 = entry probe
println!("attached kprobe: {sym}");
attached = true;
break;
}
}
if !attached {
println!("warning: no vhost/virtio symbols found in /proc/kallsyms — skipping vhost probes");
}
Ok(())
}
}attach(name, flags): The flags parameter controls whether the probe is at the function entry (0) or at the function return (>0, typically 1 for RetProbe).
Reading in userspace
#![allow(unused)]
fn main() {
// monitor/src/main.rs
use aya::maps::{HashMap, RingBuf};
use aya::programs::KProbe;
#[derive(Clone, Copy, Debug)]
#[repr(C)]
pub struct StallEvent {
pub cpu_id: u32,
pub gap_ns: u64,
}
async fn poll_vhost(stall_buf: &mut aya::maps::RingBuf) -> anyhow::Result<()> {
while let Some(item) = stall_buf.next() {
let event = unsafe {
std::ptr::read_unaligned(item.as_ptr() as *const StallEvent)
};
println!(
"vhost stall: cpu={} gap={}µs",
event.cpu_id,
event.gap_ns / 1000
);
}
Ok(())
}
}Version compatibility
This is the hardest part of kprobe-based instrumentation. Function names change, struct layouts change, and some functions are added or removed between kernel versions.
Practical strategies:
- Probe what exists: Look up symbols in
/proc/kallsymsat runtime and only attach to what’s there - Graceful degradation: If the vhost handler functions aren’t available, fall back to counting at a higher-level tracepoint (like
sched:sched_switchorirq:softirq_entry) - Per-version testing: Test your probes on multiple kernel versions
For a monitoring tool that needs broad compatibility, tracepoints are always preferable when available. KProbes are for the deep instrumentation that tracepoints can’t reach.
Next: Part 12 — Queue Depth Histograms — Build per-CPU histograms in eBPF and compute p50 and p99 percentiles.
Part 12 — Queue Depth Histograms
Raw counts are useful. But p50 and p99 tell you what the experience actually feels like.
Here’s the problem with raw counts: they don’t distinguish between a workload that’s consistently mildly queued and one that has extreme spikes. You might measure an average queue depth of 4 — but if every measurement is either 0 or 100, the average is misleading. Percentiles reveal the shape of the distribution.
The eBPF program maintains a histogram in a per-CPU array. Each CPU increments its own copy of the bucket counter. Userspace reads all the per-CPU arrays, sums them, and computes p50 and p99 from the cumulative distribution.
The key insight: histogram IS the data structure
Statistical sampling is the usual approach: sample every Nth event, store the samples, compute percentiles from the sample set. This introduces sampling error, and extreme events may not be captured at all.
The alternative: increment a counter for every event, but put it in the right bucket. The histogram IS the data. You never store individual samples.
Bucket 0: [0] → count of times the value was exactly 0
Bucket 1: [1] → count of times the value was exactly 1
Bucket 2: [2-4] → count of times the value was 2, 3, or 4
Bucket 3: [5-8] → count of times the value was 5 through 8
Bucket 4: [9-16] → count of times the value was 9 through 16
Bucket 5: [17-32] → count of times the value was 17 through 32
Bucket 6: [33-64] → count of times the value was 33 through 64
Bucket 7: [65+] → count of times the value was 65 or more
This is a logarithmic bucket scheme — wide buckets at high values. The resolution at the low end is higher because that’s where most queues spend most of their time.
Per-CPU arrays: no lost updates
The critical challenge with histogram counters: multiple CPUs can fire the same eBPF program simultaneously. If you use a regular Array<u64>, two CPUs incrementing the same bucket at the same time would both read the same value, increment it, and write it back — lost updates.
PerCpuArray solves this. Each CPU has its own independent copy of the full array. CPU 0’s copy of buckets[3] is completely separate from CPU 1’s copy. You increment your own CPU’s copy without any contention. When userspace reads the histogram, it sums all the per-CPU values for each bucket.
#![allow(unused)]
fn main() {
// ebpf-programs/src/histogram.rs
use aya_ebpf::maps::PerCpuArray;
use aya_ebpf::macros::map;
#[map]
static QUEUE_HIST: PerCpuArray<u64> = PerCpuArray::with_max_entries(8, 0);
// Bucket boundaries (queue depth)
// Bucket 0: depth=0, Bucket 1: depth=1, etc.
const BUCKET_BOUNDARIES: &[u32] = &[
1, // [0] bucket 0
2, // [1] bucket 1
5, // [2-4] bucket 2
9, // [5-8] bucket 3
17, // [9-16] bucket 4
33, // [17-32] bucket 5
65, // [33-64] bucket 6
u32::MAX, // [65+] bucket 7
];
fn bucket_for_depth(depth: u32) -> usize {
for (i, &boundary) in BUCKET_BOUNDARIES.iter().enumerate() {
if depth < boundary {
return i;
}
}
BUCKET_BOUNDARIES.len() - 1
}
#[inline(always)]
fn increment_bucket(bucket: usize) {
// get_ptr_mut returns a pointer to the current CPU's copy
if let Some(ptr) = QUEUE_HIST.get_ptr_mut(bucket as u32) {
// SAFETY: each CPU has its own independent array copy.
// No other CPU can write to this location.
unsafe {
*ptr += 1;
}
}
}
}Each CPU has its own copy of QUEUE_HIST. When CPU 0 calls increment_bucket(3), it writes to CPU 0’s copy. CPU 1’s call writes to CPU 1’s copy. No contention, no lost updates.
Integrating into the scheduler tracepoint
Attach the histogram increment to a tracepoint or kprobe that provides the metric you care about:
#![allow(unused)]
fn main() {
// ebpf-programs/src/scheduler.rs
// (continued from Part 6)
use aya_ebpf::maps::PerCpuArray;
use aya_ebpf::programs::TracePointContext;
use aya_ebpf::macros::{map, tracepoint};
#[map]
static QUEUE_HIST: PerCpuArray<u64> = PerCpuArray::with_max_entries(8, 0);
const BUCKET_BOUNDARIES: &[u32] = &[1, 2, 5, 9, 17, 33, 65, u32::MAX];
fn bucket_for_depth(depth: u32) -> usize {
for (i, &boundary) in BUCKET_BOUNDARIES.iter().enumerate() {
if depth < boundary {
return i;
}
}
BUCKET_BOUNDARIES.len() - 1
}
fn increment_bucket(bucket: usize) {
if let Some(ptr) = QUEUE_HIST.get_ptr_mut(bucket as u32) {
unsafe { *ptr += 1; }
}
}
// Runqueue length is tracked in a hash map keyed by CPU.
// We update it on sched_waking (task about to wake) and sched_switch (task starts
// running), then emit the queue depth on each context switch.
#[map]
static RUNQUEUE_DEPTH: aya_ebpf::maps::HashMap<u32, u32> =
aya_ebpf::maps::HashMap::with_max_entries(256, 0);
// sched:sched_waking — task is about to be woken (increment runqueue depth)
// Payload: comm (char[16]) at 0, pid (u32) at 16, prio (u32) at 20, target_cpu (u32) at 24
#[tracepoint]
pub fn sched_waking_depth(ctx: TracePointContext) -> u32 {
let pid = unsafe { ctx.read_at::<i32>(16).unwrap_or(0) };
if pid > 0 {
// target_cpu tells us which CPU the task will run on
let target_cpu = unsafe { ctx.read_at::<u32>(24).unwrap_or(0) };
unsafe {
let depth = RUNQUEUE_DEPTH.get(&target_cpu).copied().unwrap_or(0u32);
let new_depth = depth + 1;
let _ = RUNQUEUE_DEPTH.insert(&target_cpu, &new_depth, 0);
}
}
0
}
// sched:sched_switch — task starts running (decrement runqueue depth, record histogram)
// Payload: prev_comm (char[16]) at 0, prev_pid (u32) at 16, prev_prio (u32) at 20,
// prev_state (u64) at 24, next_comm (char[16]) at 32, next_pid (u32) at 48
#[tracepoint]
pub fn sched_switch_depth(ctx: TracePointContext) -> u32 {
let next_pid = unsafe { ctx.read_at::<i32>(48).unwrap_or(0) };
if next_pid > 0 {
let cpu = unsafe { aya_ebpf::helpers::bpf_get_smp_processor_id() };
unsafe {
let depth = RUNQUEUE_DEPTH.get(&cpu).copied().unwrap_or(0u32);
if depth > 0 {
increment_bucket(bucket_for_depth(depth));
let new_depth = depth - 1;
let _ = RUNQUEUE_DEPTH.insert(&cpu, &new_depth, 0);
}
}
}
0
}
}When a task is dequeued from the runqueue, we read the current runqueue depth, find the right bucket, and increment it. The PerCpuArray ensures no updates are lost even under heavy scheduler activity.
This is an approximation, not a precise count. The scheduler’s actual runqueue depth is maintained by the kernel’s internal per-CPU rq struct — we can’t read that from eBPF without a kprobe on an internal function. Our approach uses sched_waking (increment) and sched_switch (decrement) as proxies for enqueue and dequeue. This can diverge from the kernel’s count in two cases: a task migrated to a different CPU between waking and running (our depth for the original CPU overcounts), or a task woken on a CPU that’s already running that same task (the sched_switch fires normally, but the depth was never incremented because the task was already on the runqueue). For histogram purposes — seeing the shape of the distribution — these small divergences don’t matter. For an exact count, you’d need to read rq->nr_running directly (via kprobe on a function that holds the runqueue lock).
Userspace reader
Userspace reads the per-CPU arrays and sums them:
#![allow(unused)]
fn main() {
use aya::maps::{PerCpuArray, PerCpuValues};
pub struct Bucket {
pub range_start: u32,
pub range_end: u32,
pub count: u64,
}
pub struct Histogram {
pub buckets: Vec<Bucket>,
pub total: u64,
pub p50: f64,
pub p99: f64,
}
const BOUNDARIES: [u32; 8] = [1, 2, 5, 9, 17, 33, 65, u32::MAX];
fn read_histogram(ebpf: &mut aya::Ebpf) -> anyhow::Result<Histogram> {
let hist: PerCpuArray<u64> = PerCpuArray::try_from(ebpf.map_mut("queue_hist")?)?;
let mut bucket_counts = vec![0u64; 8];
// Read all 8 buckets, summing across all CPUs
for idx in 0..8u32 {
let per_cpu_values: PerCpuValues<u64> = hist.get(&idx, 0)?;
bucket_counts[idx as usize] = per_cpu_values.iter().sum();
}
let mut buckets = Vec::new();
for i in 0..8 {
buckets.push(Bucket {
range_start: if i == 0 { 0 } else { BOUNDARIES[i - 1] },
range_end: BOUNDARIES[i],
count: bucket_counts[i],
});
}
let total: u64 = bucket_counts.iter().sum();
let (p50, p99) = compute_percentiles(&bucket_counts, &BOUNDARIES, total);
Ok(Histogram {
buckets,
total,
p50,
p99,
})
}
}A note on reading PerCpuArray from userspace. The PerCpuArray::get(&index, flags) method returns PerCpuValues<V> — one value per CPU, all in one call. PerCpuValues derefs to Box<[V]>, so you can call .iter() on it directly and sum across CPUs, as the read_histogram function above does.
Computing p50 and p99
The percentile is the value below which p * total events fall. From a histogram, we walk the buckets cumulatively:
#![allow(unused)]
fn main() {
fn compute_percentiles(
counts: &[u64],
boundaries: &[u32; 8],
total: u64,
) -> (f64, f64) {
if total == 0 {
return (0.0, 0.0);
}
let p50_target = (total as f64 * 0.50) as u64;
let p99_target = (total as f64 * 0.99) as u64;
let mut cumsum = 0u64;
let mut prev_cumsum = 0u64;
let mut p50 = f64::NAN;
let mut p99 = f64::NAN;
for (i, &count) in counts.iter().enumerate() {
prev_cumsum = cumsum;
cumsum += count;
if p50.is_nan() && cumsum >= p50_target {
p50 = interpolate(&boundaries, i, prev_cumsum, cumsum, p50_target);
}
if p99.is_nan() && cumsum >= p99_target {
p99 = interpolate(&boundaries, i, prev_cumsum, cumsum, p99_target);
}
if !p50.is_nan() && !p99.is_nan() {
break;
}
}
(p50, p99)
}
fn interpolate(
boundaries: &[u32; 8],
bucket_idx: usize,
prev_cumsum: u64, // cumulative count before this bucket
cumsum: u64, // cumulative count including this bucket
target: u64, // the percentile target value
) -> f64 {
// Linear interpolation: where within this bucket does the target fall?
let count_in_bucket = cumsum - prev_cumsum;
let offset_in_bucket = target.saturating_sub(prev_cumsum);
let fraction = if count_in_bucket > 0 {
offset_in_bucket as f64 / count_in_bucket as f64
} else {
0.0
};
let bucket_start = if bucket_idx == 0 { 0 } else { boundaries[bucket_idx - 1] } as f64;
let bucket_end = if boundaries[bucket_idx] == u32::MAX {
bucket_start + 1.0 // open-ended bucket: approximate as one past start
} else {
boundaries[bucket_idx] as f64
};
bucket_start + fraction * (bucket_end - bucket_start)
}
}The p50 of 2.5 and p99 of 18.3 from the example above tell you: most observations are low, but occasionally there are spikes. Without percentiles, you’d only know the average, which would hide that information.
Formatted output
#![allow(unused)]
fn main() {
fn format_histogram(hist: &Histogram) -> String {
let mut lines = Vec::new();
lines.push(format!(
"total={} p50={:.1} p99={:.1}",
hist.total, hist.p50, hist.p99
));
for bucket in &hist.buckets {
let pct = if hist.total > 0 {
(bucket.count as f64 / hist.total as f64 * 100.0).round() as i32
} else {
0
};
let range = if bucket.range_end == u32::MAX {
format!("{}+", bucket.range_start)
} else {
format!("{}-{}", bucket.range_start, bucket.range_end)
};
lines.push(format!(" {}: {:>8} {:>4}%", range, bucket.count, pct));
}
lines.join("\n")
}
}Example output for runqueue depth:
total=4821 p50=2.5 p99=18.3
0: 1200 25%
1: 980 20%
2-4: 1400 29%
5-8: 800 17%
9-16: 400 8%
17-32: 30 1%
33-64: 10 0%
65+: 1 0%
A note on atomic operations
For a regular Array<u64> (not per-CPU), you’d need atomic operations to avoid lost updates. The BPF helper for this is __atomic_fetch_add — available in Linux 5.8+:
#![allow(unused)]
fn main() {
// For non-per-CPU arrays, use BPF atomic operations
unsafe {
let ptr = QUEUE_HIST.get_ptr_mut(bucket as u32)?;
// __atomic_fetch_add(ptr, 1, __ATOMIC_RELAXED);
// Note: Aya doesn't wrap this helper directly.
// Use PerCpuArray instead — it's simpler and faster.
}
}PerCPU arrays are the preferred approach in Aya. They’re simpler, faster (no atomic operations needed), and the pattern works everywhere.
Summary
Here’s the full histogram pattern:
- eBPF: declare a
PerCpuArray<u64>withNentries (one per bucket) - eBPF: on each event, increment the correct bucket using
get_ptr_mut()+ dereference - Userspace: read all
Nentries from the per-CPU array - Userspace: sum per-CPU values for each bucket
- Userspace: compute cumulative distribution, interpolate p50 and p99
No statistical sampling. Every event is counted. The histogram IS the complete dataset.
This completes the detailed instrumentation chapters. Parts 1–2 covered the architecture and setup. Parts 3–9 covered each data source. Parts 10–12 covered block I/O, vhost rings, and histograms.
The three sources — hardware PMCs, kernel tracepoints, and procfs/sysfs — each have a different rhythm. PMCs are per-cycle counters that you poll at whatever interval suits your dashboard. eBPF programs push events into ring buffers the moment they happen. Procfs and sysfs are snapshots you read on a timer. The remaining work — wiring everything into a single polling loop, adding configuration, and shipping structured output — is engineering integration. The data sources are in place. The instrumentation works. What’s left is assembly.
Part 13 — The Integration: One Event Loop, Three Sources
You’ve built twelve instruments. Each one works on its own. Now the question becomes: how do you run them all at once without turning your monitoring system into the thing it’s supposed to observe?
The answer is a single Tokio event loop that polls all three data sources — PMC file descriptors, eBPF ring buffers, and procfs/sysfs files — on their own schedules. Tokio’s select! macro makes this natural: each source gets its own future, and select! runs them concurrently, waking whenever any source has data ready.
This part doesn’t introduce new instrumentation. It wires together what Parts 3–12 built. The reward is a single binary that prints structured metrics every second — PMCs, scheduler events, NUMA stats, thermal readings, block I/O, and histograms — all from one process.
The architecture, revisited
In Part 1, we showed the hybrid architecture as a diagram. Here’s the same architecture, but now with the concrete components we’ve built filled in:
┌─────────────────────────────────────────────────────────────────┐
│ userspace (monitor binary) │
│ │
│ tokio::select! { │
│ │
│ ┌──────────────────┐ ┌────────────────────┐ ┌────────────┐ │
│ │ PMC poller │ │ ring buffer reader │ │ file │ │
│ │ (Parts 3-5, 8) │ │ (Parts 6, 10-12) │ │ poller │ │
│ │ │ │ │ │ (7, 9) │ │
│ │ Every 1s: │ │ Whenever data │ │ Every 5s: │ │
│ │ read_counter() │ │ arrives in ring_buf │ │ read │ │
│ │ on each PMC fd │ │ .next() │ │ /sys, │ │
│ │ │ │ │ │ /proc │ │
│ └────────┬─────────┘ └──────────┬──────────┘ └─────┬──────┘ │
│ │ │ │ │
│ └───────────┬───────────┘ │ │
│ │ │ │
│ ┌────────▼────────┐ │ │
│ │ metrics sink │◄──────────────────────┘ │
│ │ (aggregate + │ │
│ │ emit JSON) │ │
│ └─────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│ │ │
perf_event_open() eBPF maps file I/O
▲ ▲ ▲
│ │ │
┌──────────┼────────────────┼────────────────┼───────────────────┐
│ │ Linux Kernel │ │ │
│ ┌───────┴────────┐ │ │ │
│ │ PMC counters │ │ │ │
│ │ (fd per event │ │ │ │
│ │ per CPU) │ │ │ │
│ └────────────────┘ │ │ │
│ │ │ │
│ ┌────────────────────────┴──────────────┐ │ │
│ │ eBPF programs │ │ │
│ │ scheduler (Part 6) │ │ │
│ │ block I/O (Part 10) │ │ │
│ │ vhost/virtio (Part 11) │ │ │
│ │ histograms (Part 12) │ │ │
│ └───────────────────────────────────────┘ │ │
│ │ │
│ ┌────────────────┐ ┌────────────────────┴──────────────┐ │
│ │ /proc/vmstat │ │/sys/class/thermal/ │ │
│ │ /proc/stat │ │/sys/devices/system/node/ │ │
│ └────────────────┘ └──────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────┘
The three sources have three different rhythms:
- PMC counters are polled on a timer. You read the file descriptor, compute a rate (delta / interval), and reset. Every 1 second is a common choice.
- eBPF ring buffers are push from the kernel’s side, but polled from userspace. The kernel writes events as they happen; userspace calls
ring_buf.next()to drain them. aya’sRingBuf::next()is non-blocking — it returnsNoneimmediately when the buffer is empty. To integrate withtokio::select!, we poll on a short timer (every 100 ms) and drain batches at each tick. - procfs/sysfs are polled on a slower timer. Thermal zones and NUMA stats don’t change every second — every 5 or 10 seconds is plenty.
The event loop needs to accommodate all three rhythms without any one source blocking the others. That’s what tokio::select! does.
The main event loop
// monitor/src/main.rs
use anyhow::Result;
use std::time::Duration;
use tokio::time;
mod metrics;
mod numa;
mod pmc;
mod thermal;
/// How often to poll PMC counters
const PMC_INTERVAL: Duration = Duration::from_secs(1);
/// How often to poll procfs/sysfs
const FILE_INTERVAL: Duration = Duration::from_secs(5);
/// Maximum events to drain from the ring buffer per tick
const RINGBUF_BATCH_SIZE: usize = 256;
/// How often to poll the eBPF ring buffer
const RINGBUF_INTERVAL: Duration = Duration::from_millis(100);
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
// Load eBPF programs and attach tracepoints
let mut ebpf = load_and_attach_ebpf()?;
// Open PMC file descriptors for the events we care about
let pmc_fds = pmc::open_counters()?;
// Read initial counter values so the first delta is meaningful
let mut prev_pmc = pmc::read_all_counters(&pmc_fds)?;
// Timer ticks for each source
let mut pmc_tick = time::interval(PMC_INTERVAL);
let mut file_tick = time::interval(FILE_INTERVAL);
// Ring buffer for eBPF events
let map = ebpf
.map_mut("events")
.ok_or_else(|| anyhow::anyhow!("events map not found"))?;
let mut ring_buf = aya::maps::RingBuf::try_from(map)?;
// Timer for ring buffer polling
let mut ringbuf_tick = time::interval(RINGBUF_INTERVAL);
// Aggregated metrics for the current interval
let mut metrics = metrics::Metrics::new();
loop {
tokio::select! {
_ = pmc_tick.tick() => {
let curr = pmc::read_all_counters(&pmc_fds)?;
let deltas = pmc::compute_deltas(&prev_pmc, &curr);
prev_pmc = curr;
metrics.update_pmc(&deltas);
}
_ = ringbuf_tick.tick() => {
for event in drain_ringbuf(&mut ring_buf, RINGBUF_BATCH_SIZE) {
metrics.update_ebpf(&event);
}
}
_ = file_tick.tick() => {
let thermal = thermal::read_all_thermal_zones()?;
let numa = numa::read_numa_stats()?;
metrics.update_thermal(&thermal);
metrics.update_numa(&numa);
}
}
// Emit metrics whenever the PMC tick fires (every 1s)
if pmc_tick.tick().is_completed() {
let output = metrics.format_output();
println!("{}", output);
metrics.reset();
}
}
}PMC helpers
The pmc module needs a few helper types and functions that the event loop uses. Here is a self-contained pmc.rs that includes everything needed for the integration:
#![allow(unused)]
fn main() {
// monitor/src/pmc.rs
use anyhow::Result;
use std::io;
/// Delta between two PMC counter samples.
#[derive(Debug, Default)]
pub struct CounterDeltas {
pub instructions: u64,
pub cycles: u64,
pub cache_references: u64,
pub cache_misses: u64,
pub branch_misses: u64,
}
/// A PMC event specification.
pub struct EventSpec {
pub name: String,
pub perf_type: u32,
pub config: u64,
}
/// Open counters for all available events (used in the first example above).
pub fn open_counters() -> io::Result<Vec<(String, i32)>> {
let mut fds = Vec::new();
for spec in available_events() {
let fd = open_pmc(spec.perf_type, spec.config, 0, -1)?;
fds.push((spec.name, fd));
}
Ok(fds)
}
/// Read all counters at once.
pub fn read_all_counters(fds: &[(String, i32)]) -> io::Result<Vec<(String, u64)>> {
let mut vals = Vec::new();
for (name, fd) in fds {
let mut val: u64 = 0;
let n = unsafe { libc::read(*fd, &mut val as *mut _ as *mut libc::c_void, 8) };
if n < 0 {
return Err(io::Error::last_os_error());
}
vals.push((name.clone(), val));
}
Ok(vals)
}
/// Compute per-second deltas between two counter readings.
pub fn compute_deltas(
prev: &[(String, u64)],
curr: &[(String, u64)],
) -> CounterDeltas {
let get = |name: &str, samples: &[(String, u64)]| -> u64 {
samples.iter().find(|(n, _)| n == name).map(|(_, v)| *v).unwrap_or(0)
};
CounterDeltas {
instructions: get("instructions", curr).saturating_sub(get("instructions", prev)),
cycles: get("cycles", curr).saturating_sub(get("cycles", prev)),
cache_references: get("cache_references", curr).saturating_sub(get("cache_references", prev)),
cache_misses: get("cache_misses", curr).saturating_sub(get("cache_misses", prev)),
branch_misses: get("branch_misses", curr).saturating_sub(get("branch_misses", prev)),
}
}
fn open_pmc(type_: u32, config: u64, pid: i32, cpu: i32) -> io::Result<i32> {
#[repr(C)]
struct PerfEventAttr {
type_: u32,
size: u32,
config: u64,
sample_period: u64,
sample_type: u64,
read_format: u64,
flags: u64,
}
let attr = PerfEventAttr {
type_,
size: std::mem::size_of::<PerfEventAttr>() as u32,
config,
sample_period: 0,
sample_type: 0,
read_format: 0,
flags: 0b101, // disabled=1, pinned=1
};
let fd = unsafe {
libc::syscall(libc::SYS_perf_event_open, &attr as *const _, pid, cpu, -1, 0)
};
if fd < 0 {
return Err(io::Error::last_os_error());
}
Ok(fd as i32)
}
fn available_events() -> Vec<EventSpec> {
vec![
EventSpec { name: "instructions".into(), perf_type: 0, config: 1 },
EventSpec { name: "cycles".into(), perf_type: 0, config: 0 },
EventSpec { name: "cache_references".into(), perf_type: 0, config: 3 },
EventSpec { name: "cache_misses".into(), perf_type: 0, config: 4 },
EventSpec { name: "branch_misses".into(), perf_type: 0, config: 5 },
]
}
}⚠️ The code below has two bugs. We’ll walk through them and fix both in the next section. Don’t copy this version — it’s here to show the problem.
There are two problems with this.
First, the if pmc_tick.tick().is_completed() block outside select! calls tick() a second time. That call waits for the next tick — it doesn’t check whether the previous one fired. The result is a full extra interval delay before each emission. The PMC branch inside select! already consumed the tick; calling tick() again starts a fresh wait.
Second, the ring buffer branch uses drain_one, which returns a single event. If the scheduler is producing thousands of events per second, we’d only process one per select! iteration while the PMC and file ticks wait. We need to batch.
Let’s fix both.
Batched ring buffer draining
The fix: drain the ring buffer up to a batch size on each wake, not one at a time:
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let mut ebpf = load_and_attach_ebpf()?;
let pmc_fds = pmc::open_counters()?;
let mut prev_pmc = pmc::read_all_counters(&pmc_fds)?;
let mut pmc_tick = time::interval(PMC_INTERVAL);
let mut file_tick = time::interval(FILE_INTERVAL);
let mut ringbuf_tick = time::interval(RINGBUF_INTERVAL);
let map = ebpf
.map_mut("events")
.ok_or_else(|| anyhow::anyhow!("events map not found"))?;
let mut ring_buf = aya::maps::RingBuf::try_from(map)?;
let mut metrics = metrics::Metrics::new();
loop {
tokio::select! {
_ = pmc_tick.tick() => {
let curr = pmc::read_all_counters(&pmc_fds)?;
let deltas = pmc::compute_deltas(&prev_pmc, &curr);
prev_pmc = curr;
metrics.update_pmc(&deltas);
// Emit and reset on every PMC tick
let output = metrics.format_output();
println!("{}", output);
metrics.reset();
}
_ = ringbuf_tick.tick() => {
for event in drain_ringbuf(&mut ring_buf, RINGBUF_BATCH_SIZE) {
metrics.update_ebpf(&event);
}
}
_ = file_tick.tick() => {
let thermal = thermal::read_all_thermal_zones()?;
let numa = numa::read_numa_stats()?;
metrics.update_thermal(&thermal);
metrics.update_numa(&numa);
}
}
}
}
/// Drain up to `limit` events from the ring buffer.
/// Returns immediately if the buffer is empty — doesn't block.
/// This is synchronous because aya's `RingBuf::next()` is non-blocking;
/// it returns `None` when the buffer is empty rather than waiting.
fn drain_ringbuf<T: std::borrow::Borrow<aya::maps::MapData>>(
ring_buf: &mut aya::maps::RingBuf<T>,
limit: usize,
) -> Vec<SchedulerEvent> {
let mut events = Vec::with_capacity(limit);
for _ in 0..limit {
match ring_buf.next() {
Some(item) => events.push(parse_ringbuf_item(&*item)),
None => break,
}
}
events
}drain_ringbuf is synchronous — it calls ring_buf.next() in a tight loop up to the batch limit. This is fine because ring_buf.next() is non-blocking: it returns None immediately when the buffer is empty. We call it on a timer tick (every 100 ms) rather than trying to make it a future that blocks on data arrival. aya’s RingBuf doesn’t provide an async next(), so the timer-based approach is how we integrate it with tokio::select! without busy-looping.
Why batch? Without batching, a burst of 10,000 scheduler events would require 10,000 iterations through select!, each one checking the PMC timer and file timer before getting back to the ring buffer. With batching, we drain up to 256 events in one timer tick, then yield back to select! so the other sources get their turn. The batch size is a tuning knob: too small and you waste cycles on select! overhead; too large and you starve the PMC and file polls. 256 is a reasonable starting point — adjust based on your event rate.
Why a timer, not a blocking wait? aya’s RingBuf::next() is non-blocking — it returns None when the buffer is empty, rather than sleeping until data arrives. We can’t use it directly in select! as an always-ready future (that would busy-loop). The timer approach is honest: we poll the buffer every 100 ms, drain what’s there, and move on. If aya adds an async ring buffer API in the future, you could replace the timer with a true async poll — the rest of the loop wouldn’t change.
Unified event types
The eBPF programs from Parts 2, 6, and 10 all write to the same EVENTS ring buffer. To distinguish them in userspace, we use a unified event struct with an event_type tag at the front. Your eBPF programs should set this tag before writing to the ring buffer.
#![allow(unused)]
fn main() {
// monitor/src/events.rs
/// Event discriminant written by eBPF programs.
/// Use an explicit `u32` tag instead of a C enum so that the
/// userspace parser knows the exact size and alignment.
/// C enums are implementation-defined in size (often 4 bytes on Linux,
/// but the compiler chooses). A `u32` tag removes the ambiguity.
#[derive(Clone, Copy, Debug)]
pub enum EventType {
ContextSwitch = 0,
Wakeup = 1,
BioQueue = 2,
}
/// Unified event read from the ring buffer.
///
/// The layout must match what the eBPF program writes.
/// The eBPF side writes a struct with a `u32` tag followed by a `u32` cpu_id,
/// totaling 8 bytes with natural alignment. If you change this struct,
/// you must update the eBPF side to match.
#[derive(Clone, Copy, Debug)]
#[repr(C)]
pub struct RawEvent {
pub tag: u32,
pub cpu_id: u32,
}
/// Parsed event with a typed discriminant.
pub struct SchedulerEvent {
pub event_type: EventType,
pub cpu_id: u32,
}
/// Parse a raw `RingBufItem` into a `SchedulerEvent`.
///
/// The eBPF program writes an 8-byte struct: a `u32` tag
/// followed by a `u32` cpu_id. We read both fields at their
/// natural offsets (0 and 4) rather than assuming a packed
/// 5-byte layout. If the eBPF side uses a C enum for the tag,
/// the enum is 4 bytes on Linux (not 1), so reading the tag
/// as a single byte at offset 0 would miss the upper 3 bytes
/// and read `cpu_id` from the wrong offset.
pub fn parse_ringbuf_item(item: &[u8]) -> SchedulerEvent {
// Read the 8-byte struct at its natural layout
let raw = if item.len() >= std::mem::size_of::<RawEvent>() {
let tag = u32::from_le_bytes([
item[0], item[1], item[2], item[3],
]);
let cpu_id = u32::from_le_bytes([
item[4], item[5], item[6], item[7],
]);
RawEvent { tag, cpu_id }
} else {
RawEvent { tag: 0, cpu_id: 0 }
};
let event_type = match raw.tag {
0 => EventType::ContextSwitch,
1 => EventType::Wakeup,
2 => EventType::BioQueue,
_ => EventType::ContextSwitch,
};
SchedulerEvent { event_type, cpu_id: raw.cpu_id }
}
}Loading and attaching eBPF programs
The load_and_attach_ebpf function combines what Part 2 introduced and Part 6 elaborated:
#![allow(unused)]
fn main() {
use aya::programs::TracePoint;
use aya::Ebpf;
fn load_and_attach_ebpf() -> Result<Ebpf> {
// Load the eBPF object embedded at compile time
let mut ebpf = Ebpf::load(aya::include_bytes_aligned!(
concat!(env!("OUT_DIR"), "/perf-monitor")
))?;
// Attach scheduler tracepoints (Part 6)
attach_tracepoint(&mut ebpf, "sched_switch")?;
attach_tracepoint(&mut ebpf, "sched_waking")?;
// Attach block I/O tracepoint (Part 10)
attach_tracepoint(&mut ebpf, "block_bio_queue")?;
Ok(ebpf)
}
fn attach_tracepoint(ebpf: &mut Ebpf, name: &str) -> Result<()> {
let program: &mut TracePoint = ebpf
.program_mut(name)
.ok_or_else(|| anyhow::anyhow!("program '{}' not found", name))?
.try_into()?;
program.load()?;
// Map program names to their tracepoint (category, name)
let (category, tp_name) = match name {
"sched_switch" => ("sched", "sched_switch"),
"sched_waking" => ("sched", "sched_waking"),
"block_bio_queue" => ("block", "block_bio_queue"),
_ => return Err(anyhow::anyhow!("unknown program: {}", name)),
};
program.attach(category, tp_name)?;
Ok(())
}
}The mapping from program name to tracepoint category and event name is explicit. You could derive it — the program name often matches the tracepoint name — but being explicit avoids surprises when the two diverge (as they do for some kprobes).
NUMA helper
The numa module provides per-node memory stats. Your earlier numa.rs from Part 7 may have a different NumaStats for vmstat. For integration we need the sysfs meminfo version:
#![allow(unused)]
fn main() {
// monitor/src/numa.rs
use std::fs;
use std::io;
use serde::Serialize;
#[derive(Debug, Default, Serialize, Clone)]
pub struct NumaMemStats {
pub nodes: Vec<NodeMemInfo>,
}
#[derive(Debug, Default, Serialize, Clone)]
pub struct NodeMemInfo {
pub node: u32,
pub total_mb: u64,
pub free_mb: u64,
}
/// Read per-node memory info from sysfs and return total / free in MiB.
pub fn read_numa_stats() -> io::Result<NumaMemStats> {
let mut nodes = Vec::new();
let node_dir = std::path::Path::new("/sys/devices/system/node");
for entry in fs::read_dir(node_dir)?.flatten() {
let name_str = entry.file_name().to_string_lossy();
if !name_str.starts_with("node") {
continue;
}
let node_id: u32 = name_str.trim_start_matches("node").parse().unwrap_or(0);
let meminfo_path = entry.path().join("meminfo");
if !meminfo_path.exists() {
continue;
}
let content = fs::read_to_string(&meminfo_path)?;
let mut total_kb: u64 = 0;
let mut free_kb: u64 = 0;
for line in content.lines() {
let mut parts = line.split_whitespace();
let field = parts.next().unwrap_or("").trim_end_matches(':');
let value: u64 = parts.next().unwrap_or("0").parse().unwrap_or(0);
match field {
"MemTotal" => total_kb = value,
"MemFree" => free_kb = value,
_ => {}
}
}
nodes.push(NodeMemInfo {
node: node_id,
total_mb: total_kb / 1024,
free_mb: free_kb / 1024,
});
}
Ok(NumaMemStats { nodes })
}
}The metrics aggregator
The Metrics struct collects data from all three sources and formats it for output. Each source updates a different field:
#![allow(unused)]
fn main() {
// monitor/src/metrics.rs
use crate::events::{EventType, SchedulerEvent};
use crate::numa::NumaMemStats;
use crate::thermal::ThermalZone;
use serde::Serialize;
#[derive(Debug, Default)]
pub struct Metrics {
pub pmc: PmcMetrics,
pub scheduler: SchedulerMetrics,
pub block_io: BlockIoMetrics,
pub numa: Option<NumaMetrics>,
pub thermal: Option<ThermalMetrics>,
pub histogram: Option<HistogramMetrics>,
}
#[derive(Debug, Default, Serialize)]
pub struct PmcMetrics {
pub instructions: u64,
pub cycles: u64,
pub cache_references: u64,
pub cache_misses: u64,
pub branch_misses: u64,
}
#[derive(Debug, Default, Serialize)]
pub struct SchedulerMetrics {
pub context_switches: u64,
pub wakeups: u64,
pub per_cpu_switches: std::collections::HashMap<u32, u64>,
}
#[derive(Debug, Default, Serialize)]
pub struct BlockIoMetrics {
pub bio_queue_events: u64,
pub total_bytes: u64,
}
#[derive(Debug, Default, Serialize)]
pub struct HistogramMetrics {
pub buckets: Vec<u64>,
pub overflow: u64,
}
#[derive(Debug, Default, Serialize)]
pub struct NumaMetrics {
pub nodes: Vec<NodeMemInfo>,
}
#[derive(Debug, Default, Serialize, Clone)]
pub struct NodeMemInfo {
pub node: u32,
pub total_mb: u64,
pub free_mb: u64,
}
#[derive(Debug, Default, Serialize)]
pub struct ThermalMetrics {
pub zones: Vec<ZoneReading>,
}
#[derive(Debug, Default, Serialize)]
pub struct ZoneReading {
pub zone: String,
pub temp_c: f64,
pub trip_c: f64,
}
impl Metrics {
pub fn new() -> Self {
Self::default()
}
pub fn update_pmc(&mut self, deltas: &pmc::CounterDeltas) {
self.pmc.instructions = deltas.instructions;
self.pmc.cycles = deltas.cycles;
self.pmc.cache_references = deltas.cache_references;
self.pmc.cache_misses = deltas.cache_misses;
self.pmc.branch_misses = deltas.branch_misses;
}
pub fn update_ebpf(&mut self, event: &SchedulerEvent) {
match event.event_type {
EventType::ContextSwitch => {
self.scheduler.context_switches += 1;
*self.scheduler.per_cpu_switches
.entry(event.cpu_id)
.or_insert(0) += 1;
}
EventType::Wakeup => {
self.scheduler.wakeups += 1;
}
EventType::BioQueue => {
self.block_io.bio_queue_events += 1;
}
}
}
pub fn update_thermal(&mut self, zones: &[thermal::ThermalZone]) {
self.thermal = Some(ThermalMetrics {
zones: zones.iter().map(|z| {
// Convert from Part 9's millidegrees to Celsius.
// Find the lowest critical trip point for the trip_c field.
let trip_c = z.trip_points.iter()
.filter(|tp| tp.trip_type == "critical")
.map(|tp| tp.temp_millicelsius as f64 / 1000.0)
.fold(f64::MAX, f64::min);
ZoneReading {
zone: z.name.clone(),
temp_c: z.temp_millicelsius as f64 / 1000.0,
trip_c,
}
}).collect(),
});
}
pub fn update_numa(&mut self, numa: &numa::NumaMemStats) {
self.numa = Some(NumaMetrics {
nodes: numa.nodes.clone(),
});
}
pub fn update_histogram(&mut self, hist: &HistogramMetrics) {
self.histogram = Some(HistogramMetrics {
buckets: hist.buckets.clone(),
overflow: hist.overflow,
});
}
pub fn reset(&mut self) {
self.pmc = PmcMetrics::default();
self.scheduler = SchedulerMetrics::default();
self.block_io = BlockIoMetrics::default();
self.histogram = None;
// Don't reset numa and thermal — they're point-in-time snapshots,
// not interval counters. They keep their last-read values until
// the next file poll overwrites them.
}
pub fn format_output(&self) -> String {
let ipc = if self.pmc.cycles > 0 {
self.pmc.instructions as f64 / self.pmc.cycles as f64
} else {
0.0
};
let cache_miss_rate = if self.pmc.cache_references > 0 {
self.pmc.cache_misses as f64 / self.pmc.cache_references as f64 * 100.0
} else {
0.0
};
let branch_miss_rate = if self.pmc.instructions > 0 {
self.pmc.branch_misses as f64 / self.pmc.instructions as f64 * 100.0
} else {
0.0
};
let mut lines = Vec::new();
lines.push(format!(
"IPC={:.2} cache_miss={:.1}% branch_miss={:.3}% switches={} wakeups={}",
ipc, cache_miss_rate, branch_miss_rate,
self.scheduler.context_switches,
self.scheduler.wakeups,
));
if self.block_io.bio_queue_events > 0 {
lines.push(format!(
" bio_queue={} bio_bytes={}",
self.block_io.bio_queue_events,
self.block_io.total_bytes,
));
}
if let Some(ref hist) = self.histogram {
lines.push(format!(
" histogram: overflow={} buckets={}",
hist.overflow,
hist.buckets.iter().sum::<u64>(),
));
}
if let Some(ref numa) = self.numa {
for node in &numa.nodes {
lines.push(format!(
" node{}: {}/{} MB free",
node.node, node.free_mb, node.total_mb,
));
}
}
if let Some(ref thermal) = self.thermal {
for zone in &thermal.zones {
lines.push(format!(
" {}: {:.0}°C (trip: {:.0}°C)",
zone.zone, zone.temp_c, zone.trip_c,
));
}
}
lines.join("\n")
}
}
}Why reset() doesn’t clear thermal and NUMA. PMC counters and scheduler events are interval metrics — they accumulate over the reporting period and need to be zeroed. Thermal readings and NUMA memory stats are point-in-time snapshots. The last reading is still valid until the next poll replaces it. If you zeroed them on reset, the output would flicker between “last known value” and “no data” on every tick, which is confusing.
The format_output method uses conditional formatting. If there were no block I/O events in this interval, the bio_queue line is suppressed. If thermal zones haven’t been read yet (first 5 seconds), no thermal lines appear. This keeps the output clean without needing a separate “disable this source” flag.
Structured output: JSON mode
Human-readable output is good for development. For production monitoring, you want structured output that a downstream consumer — Prometheus, a log aggregator, a dashboard — can parse. The serde::Serialize derive on the metric types makes this straightforward:
#![allow(unused)]
fn main() {
impl Metrics {
/// Format metrics as a single JSON object.
/// Includes a timestamp so the consumer can align
/// readings from different sources.
pub fn format_json(&self) -> Result<String> {
#[derive(Serialize)]
struct Output<'a> {
timestamp: String,
pmc: &'a PmcMetrics,
scheduler: &'a SchedulerMetrics,
block_io: &'a BlockIoMetrics,
numa: &'a Option<NumaMetrics>,
thermal: &'a Option<ThermalMetrics>,
}
let output = Output {
timestamp: chrono::Utc::now().to_rfc3339(),
pmc: &self.pmc,
scheduler: &self.scheduler,
block_io: &self.block_io,
numa: &self.numa,
thermal: &self.thermal,
};
Ok(serde_json::to_string(&output)?)
}
}
}Add a CLI flag to choose the format:
// monitor/src/main.rs
use clap::Parser;
#[derive(Clone, clap::ValueEnum)]
enum OutputFormat {
Text,
Json,
}
#[derive(Parser)]
#[command(name = "perf-monitor")]
struct Args {
/// Output format: text (human-readable) or json (structured)
#[arg(long, default_value = "text")]
format: OutputFormat,
}
#[tokio::main]
async fn main() -> Result<()> {
let args = Args::parse();
// ... (same setup as before) ...
loop {
tokio::select! {
_ = pmc_tick.tick() => {
let curr = pmc::read_all_counters(&pmc_fds)?;
let deltas = pmc::compute_deltas(&prev_pmc, &curr);
prev_pmc = curr;
metrics.update_pmc(&deltas);
// Emit and reset on every PMC tick (inside the branch,
// not after select! — see the earlier "double-tick" fix)
let output = match args.format {
OutputFormat::Text => metrics.format_output(),
OutputFormat::Json => metrics.format_json()?,
};
println!("{}", output);
metrics.reset();
}
_ = ringbuf_tick.tick() => {
// ring_buf was created once before the loop —
// just drain it on each tick
for event in drain_ringbuf(&mut ring_buf, RINGBUF_BATCH_SIZE) {
metrics.update_ebpf(&event);
}
}
_ = file_tick.tick() => {
if let Ok(zones) = thermal::read_all_thermal_zones() {
metrics.update_thermal(&zones);
}
if let Ok(numa) = numa::read_numa_stats() {
metrics.update_numa(&numa);
}
}
}
}
}This adds clap and serde_json to Cargo.toml:
[dependencies]
aya = { version = "0.13", features = ["async_tokio"] }
tokio = { version = "1", features = ["full"] }
anyhow = "1"
clap = { version = "4", features = ["derive"] }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tracing = "0.1"
tracing-subscriber = "0.3"
chrono = "0.4"
Configuration: What to poll and how often
Hard-coded intervals are fine for a tutorial. For a real tool, you want the intervals to be configurable without recompiling. A simple config file works:
# perf-monitor.toml
[pmc]
interval_secs = 1
events = ["instructions", "cycles", "cache_references", "cache_misses", "branch_misses"]
[scheduler]
enabled = true
[block_io]
enabled = true
[file_poll]
interval_secs = 5
[output]
format = "text" # "text" or "json"
Reading it:
#![allow(unused)]
fn main() {
use serde::Deserialize;
#[derive(Debug, Deserialize)]
struct Config {
pmc: PmcConfig,
scheduler: SchedulerConfig,
block_io: BlockIoConfig,
file_poll: FilePollConfig,
output: OutputConfig,
}
#[derive(Debug, Deserialize)]
struct PmcConfig {
interval_secs: u64,
events: Vec<String>,
}
#[derive(Debug, Deserialize)]
struct SchedulerConfig {
enabled: bool,
}
#[derive(Debug, Deserialize)]
struct BlockIoConfig {
enabled: bool,
}
#[derive(Debug, Deserialize)]
struct FilePollConfig {
interval_secs: u64,
}
#[derive(Debug, Deserialize)]
struct OutputConfig {
format: String,
}
fn load_config(path: &std::path::Path) -> Result<Config> {
let text = std::fs::read_to_string(path)?;
let config: Config = toml::from_str(&text)?;
Ok(config)
}
}The config file controls which eBPF programs get attached (if scheduler.enabled = false, don’t attach sched_switch and sched_waking), which PMC events to open, and the polling intervals. This means you can run the same binary on a development machine (where you only care about scheduler events) and a production server (where you want everything) by switching config files.
Why TOML? It’s the lingua franca of Rust configuration. serde + toml is a single dependency, and the format is readable by humans and scripts alike. You could use JSON, but then you’d need to explain the schema somewhere — TOML’s section headers serve as built-in documentation.
Add toml to Cargo.toml:
toml = "0.8"
Conditional program attachment
With the config loaded, program attachment becomes conditional:
#![allow(unused)]
fn main() {
fn load_and_attach_ebpf(config: &Config) -> Result<Ebpf> {
let mut ebpf = Ebpf::load(aya::include_bytes_aligned!(
concat!(env!("OUT_DIR"), "/perf-monitor")
))?;
if config.scheduler.enabled {
attach_tracepoint(&mut ebpf, "sched_switch")?;
attach_tracepoint(&mut ebpf, "sched_waking")?;
}
if config.block_io.enabled {
attach_tracepoint(&mut ebpf, "block_bio_queue")?;
}
Ok(ebpf)
}
}When a source is disabled, its eBPF program is never loaded or attached. The kernel doesn’t fire it. The ring buffer receives no events for it. The update_ebpf method never sees those event types. The output never mentions them. Zero overhead for disabled sources — not just “not printed,” but “not running.”
PMC event selection from config
The PMC reader opens one file descriptor per event per CPU. Opening events you don’t need wastes file descriptors and scheduler time. The config file controls which events get opened:
#![allow(unused)]
fn main() {
// monitor/src/pmc.rs
use anyhow::{anyhow, Result};
/// A PMC event specification parsed from the config file.
pub struct EventSpec {
pub name: String,
pub perf_type: u32,
pub config: u64,
}
/// Open file descriptors for the events listed in the config.
pub fn open_counters_for(events: &[String]) -> Result<Vec<(String, i32)>> {
let all_events = available_events();
let mut fds = Vec::new();
for name in events {
let spec = all_events.iter()
.find(|e| e.name == *name)
.ok_or_else(|| anyhow!("unknown PMC event: {}", name))?;
let fd = open_pmc(spec.perf_type, spec.config, 0, -1)?;
fds.push((name.clone(), fd));
}
Ok(fds)
}
/// The built-in events this monitor supports.
/// Part 4's CPU detection code would extend this
/// with microarchitecture-specific raw events.
fn available_events() -> Vec<EventSpec> {
vec![
EventSpec { name: "instructions".into(), perf_type: 0, config: 1 },
EventSpec { name: "cycles".into(), perf_type: 0, config: 0 },
EventSpec { name: "cache_references".into(), perf_type: 0, config: 3 },
EventSpec { name: "cache_misses".into(), perf_type: 0, config: 4 },
EventSpec { name: "branch_misses".into(), perf_type: 0, config: 5 },
// Raw events (Part 4-5) would be added here based on
// the detected CPU microarchitecture:
// EventSpec { name: "llc_misses".into(), perf_type: 4, config: 0x2e_412e },
// EventSpec { name: "dtlb_walks".into(), perf_type: 4, config: 0x4f_01 },
]
}
}The available_events() function returns a static list of hardware events. Part 4’s CPU detection would extend this with raw PMC events for the detected microarchitecture (LLC misses, TLB walks, etc.). The key design point: the event list is a data structure, not a compile-time constant. The config file selects from it at runtime.
Putting it all together
The complete main.rs with configuration, conditional attachment, and structured output:
// monitor/src/main.rs
use anyhow::Result;
use clap::Parser;
use std::path::PathBuf;
use std::time::Duration;
use tokio::time;
mod metrics;
mod numa;
mod pmc;
mod thermal;
const RINGBUF_BATCH_SIZE: usize = 256;
const RINGBUF_INTERVAL: Duration = Duration::from_millis(100);
#[derive(Clone, clap::ValueEnum)]
enum OutputFormat {
Text,
Json,
}
#[derive(Parser)]
#[command(name = "perf-monitor")]
struct Args {
/// Path to config file
#[arg(short, long, default_value = "perf-monitor.toml")]
config: PathBuf,
/// Output format: text or json (overrides config file)
#[arg(long)]
format: Option<String>,
}
#[tokio::main]
async fn main() -> Result<()> {
tracing_subscriber::fmt::init();
let args = Args::parse();
let config = load_config(&args.config)?;
let output_format = match args.format.as_deref().unwrap_or(&config.output.format) {
"json" => OutputFormat::Json,
_ => OutputFormat::Text,
};
// Load and attach eBPF programs (conditional on config)
let mut ebpf = load_and_attach_ebpf(&config)?;
// Open PMC counters (only the events the config requests)
let pmc_fds = pmc::open_counters_for(&config.pmc.events)?;
let mut prev_pmc = pmc::read_all_counters(&pmc_fds)?;
// Timers
let pmc_interval = Duration::from_secs(config.pmc.interval_secs);
let file_interval = Duration::from_secs(config.file_poll.interval_secs);
let mut pmc_tick = time::interval(pmc_interval);
let mut file_tick = time::interval(file_interval);
let mut ringbuf_tick = time::interval(RINGBUF_INTERVAL);
// Ring buffer for eBPF events (created once, drained on each tick)
let mut ring_buf = {
let map = ebpf
.take_map("events")
.ok_or_else(|| anyhow::anyhow!("events map not found"))?;
aya::maps::RingBuf::try_from(map)?
};
let mut metrics = metrics::Metrics::new();
loop {
// Build the select! branches conditionally.
// If there's no ring buffer, we skip that branch.
tokio::select! {
_ = pmc_tick.tick() => {
let curr = pmc::read_all_counters(&pmc_fds)?;
let deltas = pmc::compute_deltas(&prev_pmc, &curr);
prev_pmc = curr;
metrics.update_pmc(&deltas);
// Read histograms from eBPF maps (Part 12)
if config.scheduler.enabled {
if let Ok(hist) = read_histogram(&mut ebpf) {
metrics.update_histogram(&hist);
}
}
// Emit and reset on every PMC tick
let output = match output_format {
OutputFormat::Text => metrics.format_output(),
OutputFormat::Json => metrics.format_json()?,
};
println!("{}", output);
metrics.reset();
}
_ = ringbuf_tick.tick() => {
if config.scheduler.enabled || config.block_io.enabled {
for event in drain_ringbuf(&mut ring_buf, RINGBUF_BATCH_SIZE) {
metrics.update_ebpf(&event);
}
}
}
_ = file_tick.tick() => {
if let Ok(zones) = thermal::read_all_thermal_zones() {
metrics.update_thermal(&zones);
}
if let Ok(numa) = numa::read_numa_stats() {
metrics.update_numa(&numa);
}
}
}
}
}
fn drain_ringbuf<T: std::borrow::Borrow<aya::maps::MapData>>(
ring_buf: &mut aya::maps::RingBuf<T>,
limit: usize,
) -> Vec<SchedulerEvent> {
let mut events = Vec::with_capacity(limit);
for _ in 0..limit {
match ring_buf.next() {
Some(item) => events.push(parse_ringbuf_item(&*item)),
None => break,
}
}
events
}The ringbuf_tick when all eBPF sources are disabled. The ring buffer is always created (it’s tied to the events map, which always exists in the eBPF object). When all eBPF sources are disabled, no programs write to the ring buffer, so drain_ringbuf returns an empty Vec on every tick. The timer still fires every 100 ms, but the cost is negligible — a single iteration through drain_ringbuf that immediately returns None from ring_buf.next() because the buffer is empty.
Example output
Running perf-monitor with default settings on a busy system:
IPC=1.84 cache_miss=3.2% branch_miss=0.412% switches=4821 wakeups=3847
bio_queue=142 bio_bytes=727040
node0: 12480/32168 MB free
node1: 9832/32168 MB free
x86_pkg_temp: 67°C (trip: 95°C)
acpitz: 64°C (trip: 95°C)
Same system, JSON mode (--format json):
{"timestamp":"2026-06-20T02:23:15Z","pmc":{"instructions":1842300,"cycles":1001200,"cache_references":923400,"cache_misses":29812,"branch_misses":7615},"scheduler":{"context_switches":4821,"wakeups":3847,"per_cpu_switches":{"0":1200,"1":1180,"2":1230,"3":1211}},"block_io":{"bio_queue_events":142,"total_bytes":727040},"numa":{"nodes":[{"node":0,"total_mb":32168,"free_mb":12480},{"node":1,"total_mb":32168,"free_mb":9832}]},"thermal":{"zones":[{"zone":"x86_pkg_temp","temp_c":67.0,"trip_c":95.0},{"zone":"acpitz","temp_c":64.0,"trip_c":95.0}]}}
Reading histograms from eBPF maps
Part 12 defined the eBPF side of queue depth histograms. In userspace, read the PerCpuArray and aggregate:
#![allow(unused)]
fn main() {
// monitor/src/main.rs (or a dedicated histogram module)
use aya::maps::{PerCpuArray, PerCpuValues};
use crate::metrics::HistogramMetrics;
const BOUNDARIES: [u32; 8] = [1, 2, 5, 9, 17, 33, 65, u32::MAX];
fn read_histogram(ebpf: &mut aya::Ebpf) -> anyhow::Result<HistogramMetrics> {
let hist: PerCpuArray<u64> = PerCpuArray::try_from(ebpf.map_mut("queue_hist")?)?;
let mut bucket_counts = vec![0u64; 8];
for idx in 0..8u32 {
let per_cpu_values: PerCpuValues<u64> = hist.get(&idx, 0)?;
bucket_counts[idx as usize] = per_cpu_values.iter().sum();
}
Ok(HistogramMetrics {
buckets: bucket_counts,
overflow: 0,
})
}
}What about histograms?
Part 12’s queue depth histograms use PerCpuArray, not RingBuf. They’re read on the PMC tick — same interval, same select! branch:
#![allow(unused)]
fn main() {
_ = pmc_tick.tick() => {
let curr = pmc::read_all_counters(&pmc_fds)?;
let deltas = pmc::compute_deltas(&prev_pmc, &curr);
prev_pmc = curr;
metrics.update_pmc(&deltas);
// Read histograms from eBPF maps (Part 12)
if config.scheduler.enabled {
if let Ok(hist) = read_histogram(&mut ebpf) {
metrics.update_histogram(&hist);
}
}
// Emit and reset
let output = match output_format {
OutputFormat::Text => metrics.format_output(),
OutputFormat::Json => metrics.format_json()?,
};
println!("{}", output);
metrics.reset();
}
}The histogram read is synchronous — you read all 8 per-CPU buckets and sum them. It takes microseconds. No need for a separate timer or a separate select! branch. It’s read alongside the PMC counters because both are “poll on a timer, read a value” — the histogram just happens to read from an eBPF map instead of a file descriptor.
Performance overhead
The whole point of this architecture is low overhead. Let’s quantify what “low” means:
PMC reading: One read() syscall per event per tick. 5 events × 1 tick/second = 5 syscalls/second. Each read() is a few microseconds. Total: ~25 μs/second, or 0.0025% of one CPU.
Ring buffer draining: Up to 256 events per batch, polled every 100 ms. ring_buf.next() is a memory read from a shared ring buffer — no syscall, no kernel transition. On a system with 5,000 scheduler events/second, that’s ~20 batches/second at the 256-event batch size. Each batch is a tight loop of memcpy + struct parsing. Total: well under 1 ms/second. The 100 ms poll interval adds 10 timer wakes/second, but each is a simple check — if the buffer is empty, the cost is near zero.
File polling: Reading /sys/class/thermal/thermal_zone*/temp and /sys/devices/system/node/node*/meminfo involves opening files, reading them, and closing them. Two thermal zones + two NUMA nodes = ~8 file reads every 5 seconds. Each read is ~100 μs. Total: ~800 μs every 5 seconds, or 0.016% of one CPU.
eBPF programs: These run in-kernel. Their overhead is proportional to the event rate. A sched_switch handler that reads three fields and writes a 24-byte event to a ring buffer takes ~1 μs per invocation. At 5,000 switches/second, that’s 5 ms/second on one CPU — 0.5%.
Total overhead: under 1% of one CPU. This is the benefit of the three-source architecture. PMC counters are read from file descriptors — no eBPF overhead. eBPF programs do in-kernel aggregation — only summaries cross the kernel/userspace boundary. File reads are infrequent. Nothing spins.
Summary
The integration chapter ties together the three data sources into a single event loop:
-
tokio::select!runs all three sources concurrently — PMC polling, ring buffer draining, and file reading — without any source blocking the others. -
Batched ring buffer draining (polled every 100 ms) prevents eBPF event bursts from starving the PMC and file polls. The batch size and poll interval are tuning knobs.
-
The
Metricsaggregator collects data from all sources, formats it as human-readable text or structured JSON, and resets interval counters on each tick. -
Configuration files control which sources are enabled, which PMC events to read, and the polling intervals — without recompiling.
-
Conditional program attachment means disabled sources have zero overhead — their eBPF programs are never loaded, their ring buffer events never arrive, their file reads never happen.
-
Histogram reads happen on the PMC tick alongside counter reads — they’re both “poll a map/file, read a value” operations, not push-based events.
-
The total overhead is under 1% of one CPU. PMC reading is a handful of syscalls. Ring buffer draining is memory copies. File polling is infrequent. eBPF programs run in-kernel with microsecond-scale handlers.
You now have a complete eBPF performance monitoring system: hardware counters, kernel tracepoints, and procfs/sysfs — all wired into a single binary with configurable sources, structured output, and minimal overhead. Parts 1–2 gave you the mental model and project structure. Parts 3–9 gave you each data source. Parts 10–12 gave you specialized instrumentation. This part gave you the assembly.
What you do with it from here depends on what you’re monitoring. The data sources are modular — add a new eBPF program, add a new update_* method, add a new config section. The event loop doesn’t care how many sources there are. select! just grows another branch.