kprobe (8)

kprobe [-hv] [-d secs] [-p PIDs] kprobe_definition [filter]

This will create, trace, then destroy a given kprobe. See Documentation/trace/kprobetrace.txt in the Linux kernel source for the syntax of a kprobe definition.

WARNING: This uses kernel dynamic tracing, which can cause cause kernel panics or freezes. Test, and know what you are doing, before use.

Also beware of feedback loops: tracing tcp functions over an ssh session, or writing ext4 functions to an ext4 file system. For the former, tcp trace data could be redirected to a file (as in the usage message). For the latter, trace to the screen or a different file system.

Since this uses ftrace, only the root user can use this tool.

FTRACE and KPROBES CONFIG, which you may already have enabled and available on recent kernels.

-d seconds

Set the duration of tracing, in seconds. Trace output will be buffered and printed at the end. This also reduces overheads by buffering in-kernel, instead of printing events as they occur.

The ftrace buffer has a fixed size per-CPU (see /sys/kernel/debug/tracing/buffer_size_kb). If you think events are missing, try increasing that size.

-h

Print usage message.

-H

Print column headers.

-v

Show the kprobe format file only (do not trace), identifying possible variables for use in a custom filter.

-p PID

Only trace kernel functions when this process ID is on-CPU.

kprobe_definition

A full kprobe definition, as documented by Documentation/trace/kprobetrace.txt in the Linux kernel source. See the EXAMPLES section.

filter

An ftrace filter definition.

These examples may need modification to match your kernel version's function names and platform's register usage. If using platform specific registers becomes too painful in practice, consider a kernel debuginfo-based tracer, which can trace variables names instead. For example, perf_events.

Trace do_sys_open() return values:

# kprobe 'r:myopen do_sys_open $retval'

Trace do_sys_open() file mode:

# kprobe 'p:myopen do_sys_open mode=%cx:u16'

Trace do_sys_open() file mode for PID 81:

# kprobe -p 81 'p:myopen do_sys_open mode=%cx:u16'

Trace do_sys_open() with filename string:

# kprobe 'p:myopen do_sys_open filename=+0(%si):string'

Trace do_sys_open() for filenames ending in "stat"

# kprobe 'p:myopen do_sys_open fn=+0(%si):string' 'fn ~ *stat '

The output format depends on the kernel version, and headings can be printed using -H. The format is the same as the ftrace function trace format, described in the kernel source under Documentation/trace/ftrace.txt.

Typical fields are:

TASK-PID

The process name (which could include dashes), a dash, and the process ID.

CPU#

The CPU ID, in brackets.

||||

Kernel state flags. For example, on Linux 3.16 these are for irqs-off, need-resched, hardirq/softirq, and preempt-depth.

TIMESTAMP

Time of event, in seconds.

FUNCTION

Kernel function name.

This can generate a lot of trace data quickly, depending on the frequency of the traced events. Such data will cause performance overheads. This also works without buffering by default, printing function events as they happen (uses trace_pipe), context switching and consuming CPU to do so. If needed, you can try the "-d secs" option, which buffers events instead, reducing overhead. If you think the buffer option is losing events, try increasing the buffer size (buffer_size_kb).

It's a good idea to use funccount(8) first, which is lower overhead, to help you select which functions you may want to trace using kprobe(8).

This is from the perf-tools collection:

• https://github.com/brendangregg/perf-tools

Also look under the examples directory for a text file containing example usage, output, and commentary for this tool.

Linux

Unstable - in development.

Brendan Gregg

functrace(8), funccount(8)