dwatch1

Usage

Overview

The dwatch utility uses dtrace(1) to display process info when a given DTrace probe point is triggered. Only the root user or users with either mdo(1) or sudo(8) (ports/security/sudo) access can run this command.

dwatch automates the process of generating DTrace scripts to coalesce trace output by date/time, process info, and [optionally] probe-specific data.

Output format without options is:

date/time uid.gid execname[pid]: psargs

For example, the command ‘dwatch BEGIN’ produces:

INFO Watching 'dtrace:::BEGIN' ...

2017 May 29 08:23:20 0.0 dtrace[60671]: dtrace -s /dev/stdin

The -F option causes dwatch to instead coalesce trace output by date/time, process info, and probe traversal.

Output format with the ‘-F’ option is:

date/time uid.gid execname[pid]: {->,<-, |} prov:mod:func:name ...

For example, the command ‘dwatch -F BEGIN’ produces:

INFO Watching 'dtrace:::BEGIN' ...

2017 May 29 21:34:41 0.0 dtrace[86593]: | dtrace:::BEGIN ...

The -R option causes dwatch to display a process tree containing the parent, grandparent, and ancestor process info.

Output format with the ‘-R’ option is:

date/time uid0.gid0 execname[pid0]: psargs0

 -+= pid3 uid3.gid3 psargs3

  \\-+= pid2 uid2.gid2 psargs2

    \\-+= pid1 uid1.gid1 psargs1

      \\-+= pid0 uid0.guid0 psargs0

For example, the command ‘dwatch -R BEGIN’ produces:

INFO Watching 'dtrace:::BEGIN' ...

2017 May 29 21:38:54 0.0 dtrace[86899]: dtrace -s /dev/stdin

 -+= 86855 604.604 -bash

  \\-+= 86857 604.604 /bin/sh /usr/sbin/dwatch -R BEGIN

    \\-+= 86897 0.0 sudo dtrace -s /dev/stdin

      \\-+= 86899 0.0 dtrace -s /dev/stdin

Of particular interest is the ability to filter using regular expressions. The ‘-g group’, ‘-p pid’, ‘-r regex’, ‘-u user’, and ‘-z regex’ options can be combined with ‘-R’ to match on parent process criteria as well as current process info.

In contrast, the ‘-j jail’, and ‘-k name’ options apply only to the current process even if ‘-R’ is given.

The ‘-E code’ option gives the ability to customize probe-specific data. For example, the command:

dwatch -E 'printf("%s", copyinstr(arg0))' chdir

displays the path argument sent to chdir(2) calls.

Profiles can be written for more complex routines and/or convenience. To list available profiles use the ‘-Q’ option. Use the ‘-X profile’ option to use a particular profile.

For example, the command ‘dwatch -X kill’ displays arguments sent to kill(2).

Options

If a probe argument does not contain colon (":") and none of ‘-P’, ‘-m’, ‘-f’, or ‘-n’ are given, the probe argument is intelligently mapped to its most-likely value. Use ‘dwatch -l name’ to see what probes will match a given name.

Multiple probes must be given as a single (quoted) argument, separated by comma and/or whitespace. Any/all arguments following said probes will be passed to dtrace(1) unmodified.

-1

Print one line per process/profile (Default; disables ‘-R’).

-B num

Maximum number of arguments to display (Default 64).

-d

Debug. Send dtrace(1) script to stdout instead of executing.

-e

Exit after compiling request but prior to enabling probes.

-E code

DTrace code for event details. If `-', read from stdin. This allows customization of what is printed after date/time and process info. By default, the name and arguments of the program triggering the probe are shown. Can be specified multiple times.

-f

Enable probes matching the specified function names.

-F

Coalesce trace output by probe.

-g group

Group filter. Only show processes matching group name/gid. This can be an awk(1) regular expression to match a numerical gid.

-j jail

Jail filter. Only show processes matching jail name/jid.

-k name

Only show processes matching name. Can also be of the format ‘name*’ to indicate “begins with”, ‘*name’ to indicate “ends with”, or ‘*name*’ to indicate “contains”. Can be specified multiple times.

-K num

Maximum directory depth to display (Default 64).

-l

List available probes on standard output and exit.

-m

Enable probes matching the specified module names.

-X profile

Load profile from DWATCH_PROFILES_PATH.

-n

Enable probes matching the specified probe names.

-N count

Exit after count matching entries (Default 0 for disabled).

-o file

Set output file. If ‘-’, the path ‘/dev/stdout’ is used.

-O cmd

Execute cmd for each event. This can be any valid sh(1) command. The environment variables ‘$TAG’ and ‘$DETAILS’ are set for the given cmd.

-p pid

Process id filter. Only show processes with matching pid. This can be an awk(1) regular expression.

-P

Enable probe matching the specified provider name.

-q

Quiet. Hide informational messages and all dtrace(1) errors.

-Q

List available profiles in DWATCH_PROFILES_PATH and exit.

-r regex

Filter. Only show blocks matching awk(1) regular expression.

-R

Show parent, grandparent, and ancestor of process.

-t test

Test clause (predicate) to limit events (Default none). Can be specified multiple times.

-T time

Timeout. The format is ‘#[smhd]’ or just ‘#’ for seconds.

-u user

User filter. Only show processes matching user name/uid. This can be an awk(1) regular expression to match a numerical UID.

-v

Verbose. Show all errors from dtrace(1).

-V

Report dwatch version on standard output and exit.

-w

Permit destructive actions (copyout*, stop, panic, etc.).

-x

Trace. Print ‘<probe-id>’ when a probe is triggered.

-y

Always treat stdout as console (enable colors/columns/etc.).

-z regex

Only show processes matching awk(1) regular expression.

Profiles

Profiles customize the data printed during events. Profiles are loaded from a colon-separated list of directories in DWATCH_PROFILES_PATH. This is an incomplete list of profiles with basic descriptions:

chmod

Print mode and path from chmod(2), lchmod(2), fchmodat(2)

coredump

Print coredump-worthy signal deliveries with a verdict of whether a core will actually be written

coredump-top

Maintain a running catalog of coredump-worthy signals by process and signal

dtmalloc

Print kernel malloc(9) and free(9) activity by malloc type, meeting a size threshold

dtmalloc-top

Maintain a running catalog of net bytes and allocation balance by malloc type (leak suspects rise)

errno

Print non-zero errno results from system calls

errno-NAME

Print system calls returning the given errno, by symbolic name or number (e.g., errno-ENOENT, errno-ENOTCAPABLE, errno-13)

hang

Print threads that were asleep longer than a threshold, as they wake, naming the sleeper and the waker

hang-top

Maintain a running catalog of long sleeps by process

io

Print disk I/O details provided by dtrace_io(4)

io-slow

Print disk I/O requests exceeding a latency threshold

ip

Print IPv4 and IPv6 details provided by dtrace_ip(4)

jail

Print jail management activity from jail(2), jail_set(2), jail_get(2), jail_attach(2), jail_remove(2)

kill

Print signal and pid from kill(2)

lock

Print kernel lock contention details provided by dtrace_lockstat(4)

lock-spin

Print kernel lock spinning details provided by dtrace_lockstat(4)

mib

Print network stack statistics counters as they increment (requires a kernel built with options KDTRACE_MIB_SDT)

namei

Print pathname resolution details provided by the vfs:namei:lookup() probes

namei-failure

Print failed pathname resolutions (also namei-enoent for ENOENT only)

nanosleep

Print requested time from nanosleep(2)

open

Print path from open(2), openat(2)

priv

Print privilege requests denied by priv_check(9) (also priv-ok for those granted)

proc

Print process execution details provided by dtrace_proc(4)

proc-signal

Print process signal details provided by dtrace_proc(4)

proc-signal-fatal

Print only signals whose default disposition is to terminate the receiving process

rw

Print buffer contents from read(2), write(2)

sched

Print CPU scheduling details provided by dtrace_sched(4)

sched-latency

Print threads that waited longer than a threshold between becoming runnable and running

slow

Print filesystem-related syscalls exceeding a latency threshold

slow-syscall

Print any syscall exceeding a latency threshold

tcp

Print TCP address/port details provided by dtrace_tcp(4)

tcp-io

Print TCP I/O details provided by dtrace_tcp(4)

tcp-retransmit

Print TCP retransmissions, retransmit timeouts, and connections dropped by retransmit exhaustion (requires a kernel built with options KDTRACE_MIB_SDT)

udp

Print UDP I/O details provided by dtrace_udp(4)

vop_create

Print filesystem paths being created by VOP_CREATE(9)

vop_lookup

Print filesystem paths being looked-up by VOP_LOOKUP(9)

vop_mkdir

Print directory paths being created by VOP_MKDIR(9)

vop_mknod

Print device node paths being created by VOP_MKNOD(9)

vop_readdir

Print directory paths being read by VOP_READDIR(9)

vop_remove

Print filesystem paths being removed by VOP_REMOVE(9)

vop_rename

Print filesystem paths being renamed by VOP_RENAME(9)

vop_rmdir

Print directory paths being removed by VOP_RMDIR(9)

vop_symlink

Print symlink paths being created by VOP_SYMLINK(9)

Environment

These environment variables affect the execution of dwatch:

DWATCH_HANG_MS

Minimum sleep duration, in milliseconds, displayed by the hang profiles (default 1000; 0 to show everything).

DWATCH_IO_MS

Minimum disk I/O request latency, in milliseconds, displayed by the io-slow profile (default 100; 0 to show everything).

DWATCH_LOCK_MS

Minimum lock holdoff duration, in milliseconds, displayed by the lock profiles (default 1; 0 to show everything).

DWATCH_MALLOC_MIN

Minimum allocation size, in bytes, displayed by the dtmalloc profiles (default 65536; 0 to show everything).

DWATCH_PROFILES_PATH

If DWATCH_PROFILES_PATH is set, dwatch searches for profiles in the colon-separated list of directories in that variable instead of the default ‘/usr/libexec/dwatch:/usr/local/libexec/dwatch’. If set to NULL, profiles are not loaded.

DWATCH_SCHED_MS

Minimum run-queue wait, in milliseconds, displayed by the sched-latency profile (default 10; 0 to show everything).

DWATCH_SLOW_MS

Minimum syscall latency, in milliseconds, displayed by the slow profiles (default 100).

Exit status

The dwatch utility exits 0 on success, and >0 if an error occurs.

Examples

Watch processes entering system CPU scheduler.

dwatch on-cpu

List available profiles, one line per profile.

dwatch -1 -Q

Do not execute dtrace(1) but display script on stdout and exit.

dwatch -d fsync

Compile and test but do not execute code generated with given probe.

dwatch -e test_probe

Print argument one being passed to each call of zfs_sync().

dwatch -E 'printf("%i", arg1)' zfs_sync

Watch all functions named ‘read’.

dwatch -f read

Watch all probe traversal.

dwatch -F :

Watch syscall probe traversal.

dwatch -F syscall

Display only processes belonging to wheel super-group.

dwatch -g wheel execve

Display only processes belonging to groups ‘daemon’ or ‘nobody’.

dwatch -g '1|65534' execve

Ignore jails, displaying only base system processes.

dwatch -j 0 execve

Display only processes running inside the jail named ‘myjail’.

dwatch -j myjail execve

Watch syscall traversal by ruby processes.

dwatch -k 'ruby*' -F syscall

Watch syscall traversal by processes containing ‘daemon’ in their name.

dwatch -k '*daemon*' -F syscall

Watch signals being passed to kill(2).

dwatch -X kill

Watch signals being passed between bash(1) (ports/shells/bash) and vi(1).

dwatch -k bash -k vi -X kill

Display a list of unique functions available.

dwatch -l -f

List available probes for functions ending in ‘read’.

dwatch -l -f '*read'

List available probes ending in “read”.

dwatch -l -r 'read$'

Display a list of unique providers.

dwatch -l -P

Watch paths being removed by VOP_REMOVE(9).

dwatch -X vop_remove

Watch the name ‘read’ instead of the function ‘read’. The dwatch selection algorithm will commonly favor the function named ‘read’ when not given a type (using ‘-P’, ‘-m’, ‘-f’, or ‘-n’) because there are more probes matching the function named ‘read’ than probes matching ‘read’ for any other type.

dwatch -n read

Display the first process to call kill(2) and then exit.

dwatch -N 1 kill

Watch processes forked by pid 1234.

dwatch -p 1234 execve

Watch processes forked by either pid 1234 or pid 5678.

dwatch -p '1234|5678' execve

Watch the provider ‘random’ instead of the function ‘random’. The dwatch selection algorithm will commonly favor the function named ‘random’ when not given a type (using ‘-P’, ‘-m’, ‘-f’, or ‘-n’) because there are more probes matching the function named ‘random’ than probes matching the provider named ‘random’.

dwatch -P random

Display available profiles matching ‘vop’.

dwatch -Q -r vop

Watch VOP_LOOKUP(9) paths containing ‘/lib/’.

dwatch -r /lib/ -X vop_lookup

Show process tree for each command as it is executed.

dwatch -R execve

Watch processes forked by pid 1234 or children thereof.

dwatch -R -p 1234 execve

Display processes calling write(2) with “nbytes” less than 10.

dwatch -t 'arg2<10' -E 'printf("%d",arg2)' write

Display write(2) buffer when “execname” is not ‘dtrace’ and “nbytes” is less than 10.

dwatch -X write -t 'execname != "dtrace" && this->nbytes < 10'

Watch ‘statfs’ for 5 minutes and exit.

dwatch -T 5m statfs

Display only processes belonging to the root super-user.

dwatch -u root execve

Display only processes belonging to users ‘daemon’ or ‘nobody’.

dwatch -u '1|65534' execve

Print version and exit.

dwatch -V

View the first 100 scheduler preemptions.

dwatch -y -N 100 preempt | less -R

Display processes matching either “mkdir” or “rmdir”.

dwatch -z '(mk|rm)dir' execve

Run a command and watch network activity only while that command runs.

dwatch -X tcp -- -c "nc -zvw10 google.com 22"

Watch open(2) and openat(2) calls only while pid 1234 is active.

dwatch -X open -- -p 1234

Watch probe traversal for a given command. Note that “-c true” is passed to dtrace(1) since it appears after the dwatch probe argument.

dwatch -F 'pid$target:::entry' -c true