Demonstrations of runqslower, the Linux eBPF/bcc version.

runqslower shows high latency scheduling times between tasks being

ready to run and them running on CPU after that. For example:

# runqslower

Tracing run queue latency higher than 10000 us

TIME COMM PID LAT(us)

04:16:32 cc1 12924 12739

04:16:32 sh 13640 12118

04:16:32 make 13639 12730

04:16:32 bash 13655 12047

04:16:32 bash 13657 12744

04:16:32 bash 13656 12880

04:16:32 sh 13660 10846

04:16:32 gcc 13663 12681

04:16:32 make 13668 10814

04:16:32 make 13670 12988

04:16:32 gcc 13677 11770

04:16:32 gcc 13678 23519

04:16:32 as 12999 20541

[...]

This shows various processes waiting for available CPU during a Linux kernel

build. By default the output contains delays for more than 10ms.

These delays can be analyzed in depth with "perf sched" tool, see:

* http://www.brendangregg.com/blog/2017-03-16/perf-sched.html

USAGE message:

# ./runqslower -h

usage: runqslower.py [-h] [-p PID] [min_us]

Trace high run queue latency

positional arguments:

min_us minimum run queue latecy to trace, in ms (default 10000)

optional arguments:

-h, --help show this help message and exit

-p PID, --pid PID trace this PID only

examples:

./runqslower # trace run queue latency higher than 10000 us (default)

./runqslower 1000 # trace run queue latency higher than 1000 us

./runqslower -p 123 # trace pid 123 only