back to hub
devref / cli
real-time linux

kernel config, scheduling, latency measurement, IRQ tuning, and PREEMPT_RT tooling for deterministic systems

note: many commands below require root or CAP_SYS_NICE. RT priorities range 1–99 (99 = highest). FIFO and RR are hard-RT policies; DEADLINE is for sporadic tasks. Click any card to copy the command.
all
kernel
scheduling
latency
irq / cpu
memory
tracing
system info
kernel & rt patch
check kernel version click to copy
uname -a
Show full kernel version, hostname, arch. Look for PREEMPT_RT in the string.
confirm RT kernel click to copy
uname -v | grep PREEMPT_RT
Confirm the running kernel has the full PREEMPT_RT patch applied.
check kernel config for RT preemption click to copy
grep -i PREEMPT /boot/config-$(uname -r)
Inspect build-time preemption model: PREEMPT_RT_FULL is the target for hard-RT workloads.
view kernel boot parameters click to copy
cat /proc/cmdline
Check active kernel boot args — useful to confirm isolcpus, nohz_full, rcu_nocbs, etc.
kernel ring buffer (recent) click to copy
dmesg -T | tail -n 50
Last 50 kernel messages with human-readable timestamps. Useful post-boot for RT config warnings.
verify nohz_full CPUs click to copy
cat /sys/devices/system/cpu/nohz_full
Shows CPUs configured for tickless operation — essential to eliminate timer interrupt jitter on RT cores.
process scheduling
run command with FIFO RT priority click to copy
chrt -f 80 <command>
Launch a process with SCHED_FIFO at priority 80. It will preempt any non-RT task.
run command with RR scheduling click to copy
chrt -r 50 <command>
SCHED_RR: round-robin among tasks at the same priority. Time-slice based.
change priority of running process click to copy
chrt -f -p 75 <PID>
Set SCHED_FIFO priority 75 on an already-running process by PID.
show scheduling policy of process click to copy
chrt -p <PID>
Print the current scheduling policy and priority of a process.
set SCHED_DEADLINE parameters click to copy
chrt -d --sched-runtime 5000000 --sched-deadline 10000000 --sched-period 10000000 0 <command>
Run with SCHED_DEADLINE: 5 ms runtime within a 10 ms deadline and 10 ms period (values in nanoseconds).
pin process to CPU core(s) click to copy
taskset -cp 2,3 <PID>
Restrict process to cores 2 and 3. Combine with chrt for RT + affinity.
launch command on specific CPU click to copy
taskset -c 3 <command>
Start a new process pinned to CPU 3 from the beginning.
set nice priority (non-RT) click to copy
nice -n -20 <command>
Run a SCHED_OTHER task at highest nice priority (-20). Still preempted by any RT task.
change nice of running process click to copy
renice -n -10 -p <PID>
Adjust nice value of a running process (requires root for negative values).
latency measurement
cyclictest — basic latency test click to copy
cyclictest -l 100000 -m -n -p 99
Run 100k iterations, mlockall, nanosleep, priority 99. The gold-standard RT latency benchmark.
cyclictest with histogram output click to copy
cyclictest -m -n -p 99 -i 1000 -h 200 -q -l 1000000
Collect a latency histogram up to 200 µs over 1M loops. Pipe to gnuplot or dump to file for analysis.
cyclictest on all CPUs (SMP) click to copy
cyclictest -S -m -n -p 99 -i 1000 -l 100000
Run one thread per CPU simultaneously — important for identifying per-core latency outliers.
latencytop — identify latency sources click to copy
latencytop
Interactive view of what's causing scheduling latency system-wide. Requires latencytop package.
enable wakeup RT latency tracer click to copy
echo wakeup_rt > /sys/kernel/debug/tracing/current_tracer
Switch ftrace to wakeup_rt mode: records max latency from RT task wakeup to execution. See also hwlat tracer.
hardware latency detector (hwlat) click to copy
echo hwlat > /sys/kernel/debug/tracing/current_tracer
Detects hardware-induced latencies (SMI, firmware, BIOS). Must rule these out before blaming the kernel.
read ftrace latency results click to copy
cat /sys/kernel/debug/tracing/trace
Dump the current trace buffer. Check max_latency file for the worst-case wakeup time recorded.
stress system during latency test click to copy
stress -c 4 -i 2 -m 2 -t 60
Generate CPU, I/O, and memory stress for 60 s while cyclictest runs. Reveals worst-case latency under load.
irq & cpu isolation
set IRQ affinity to CPU 0 only click to copy
echo 1 > /proc/irq/<IRQ>/smp_affinity
Route interrupt <IRQ> exclusively to CPU 0 (bitmask: 1 = CPU0). Keeps RT cores interrupt-free.
set IRQ affinity (list form) click to copy
echo 0 > /proc/irq/<IRQ>/smp_affinity_list
Same as above using human-readable CPU list format instead of bitmask.
stop irqbalance daemon click to copy
systemctl stop irqbalance
irqbalance will override manual affinity settings — stop it before tuning IRQ placement.
view isolated CPUs click to copy
cat /sys/devices/system/cpu/isolated
Lists CPUs isolated via isolcpus= boot param. Isolated CPUs won't receive scheduler tasks unless explicitly pinned.
check RCU callbacks offloaded CPUs click to copy
cat /sys/devices/system/cpu/rcu_nocbs
Shows CPUs where RCU callbacks are offloaded (rcu_nocbs= boot param). Required to eliminate RCU jitter on RT cores.
set CPU governor to performance click to copy
cpupower frequency-set -g performance
Disable frequency scaling — P-state transitions add determinism-killing latency. Use on all RT cores.
disable deep CPU idle states click to copy
cpupower idle-set -D 0
Disable all idle states deeper than C0. C-state exit latency can be hundreds of microseconds.
move all IRQs away from CPU 3 (tuna) click to copy
tuna --cpus=3 --isolate
tuna is a high-level RT tuning tool that moves IRQs and threads off specified CPUs in one command.
view interrupt counts per CPU click to copy
watch -n1 cat /proc/interrupts
Live view of interrupt counts per IRQ per CPU. Verify RT CPUs are receiving zero or minimal interrupts.
memory locking & hugepages
lock all process memory (in C) click to copy
mlockall(MCL_CURRENT | MCL_FUTURE)
Call in your RT application to prevent page faults — any fault during RT execution is catastrophic for latency.
check memlock limit click to copy
ulimit -l
Show max locked memory in KB. Set to unlimited in /etc/security/limits.conf for RT users.
disable transparent hugepages click to copy
echo never > /sys/kernel/mm/transparent_hugepage/enabled
THP compaction can cause multi-millisecond stalls. Disable for RT workloads; use explicit hugepages if needed.
allocate static 2 MB hugepages click to copy
echo 512 > /proc/sys/vm/nr_hugepages
Pre-allocate 512 × 2 MB = 1 GB of hugepages. Pre-allocation avoids runtime TLB pressure for large buffers.
disable swapping click to copy
sysctl -w vm.swappiness=0
A swap event on an RT task will blow any latency budget. Set to 0 or disable swap entirely.
bind process to NUMA node click to copy
numactl --cpunodebind=0 --membind=0 <command>
Run on NUMA node 0, allocating memory local to that node. Remote NUMA access adds ~50–100 ns of latency.
tracing & debugging
enable ftrace tracing click to copy
echo 1 > /sys/kernel/debug/tracing/tracing_on
Toggle the global ftrace recording switch. Set to 0 to pause without losing the buffer.
list available tracers click to copy
cat /sys/kernel/debug/tracing/available_tracers
Shows built-in tracers: nop, function, function_graph, wakeup, wakeup_rt, hwlat, etc.
record RT wakeup trace (trace-cmd) click to copy
trace-cmd record -p wakeup_rt -b 10000
Record a wakeup_rt trace with 10 MB ring buffer using trace-cmd front-end. Replay with trace-cmd report.
perf sched latency report click to copy
perf sched record -a -- sleep 5 && perf sched latency
Record 5 s of scheduling events then display per-task wakeup latency statistics.
perf stat on a process click to copy
perf stat -p <PID> -e cycles,cache-misses,context-switches
Count cycles, cache misses, and context switches on a running RT task.
trace system calls of RT process click to copy
strace -p <PID> -e trace=sched_setscheduler,nanosleep,clock_nanosleep
Monitor only the RT-relevant syscalls — useful to confirm scheduling and sleep calls behave as expected.
rtla timerlat histogram click to copy
rtla timerlat hist -u -d 30
Collect a 30-second timer latency histogram from user-space perspective. rtla ships with kernel-tools since 5.17.
rtla osnoise top click to copy
rtla osnoise top -d 10
Show OS noise (interference) summary per CPU for 10 seconds — isolates kernel-induced jitter sources.
system info & monitoring
list RT processes with policies click to copy
ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan,comm
Show all processes with scheduling class (FF/RR/TS), RT priority, and CPU assignment.
watch RT processes live click to copy
watch -n0.5 'ps -eo pid,class,rtprio,comm | grep -v TS'
Refresh every 0.5 s showing only non-SCHED_OTHER (i.e. RT) tasks and their priorities.
tuna — show thread/IRQ overview click to copy
tuna --show_threads
Display all threads with scheduling policy, priority, and affinity in one table.
CPU topology overview click to copy
lscpu
Show socket/core/thread topology, cache sizes, NUMA config. Essential for planning CPU isolation layout.
visualise NUMA/CPU topology click to copy
lstopo --of ascii
ASCII diagram of the full hardware topology (hwloc package). Helps plan NUMA-aware affinity.
check kernel timer frequency (HZ) click to copy
grep CONFIG_HZ= /boot/config-$(uname -r)
RT kernels are typically built at HZ=1000 (1 ms tick). Lower HZ means coarser sleep granularity.
live context switches & interrupts click to copy
vmstat 1
Column cs = context switches/s, in = interrupts/s. Both should be minimal on isolated RT CPUs.
monitor CPU C-states & frequency click to copy
turbostat --interval 1 --quiet
Per-CPU frequency, C-state residency, and power. Verify RT cores stay at full frequency in C0.
copied!