“Finding origins of latencies using Ftrace
“Ftrace developer Steven Rostedt describes how to use ftrace to
find unexpected and/or unacceptable latencies, which may be a
barrier to realtime processing, in his paper [PDF]. Ftrace is a
relatively new tool in the kernel that provides various kinds of
tracing information and has some facilities that can be used
specifically for tracking down latency issues. Tracers like
irqsoff, preemptoff, and wakeup (along with some variants) capture
information while the kernel is running in specific modes (i.e.
with interrupts disabled, preemption turned off, etc.).“Rostedt’s paper gives a fairly detailed look at the tracers,
how to enable them, what they do, and the output they produce.
While these latency tracers are active, they capture things like
kernel functions called or trace event points encountered while
looking for the maximum time spent in the latency-causing modes. By
looking at what the kernel is doing when the latency has exceeded
expectations, it can lead a developer to the specific
cause—which may lead to a way to reduce the latency. Rostedt
mentions the JACK “audio connection kit” developers as an early
adopter of latency tracing, noting that they found both kernel and
JACK bugs that were causing excess latency.”