The definitive system and process troubleshooting tool
sysdig [option]... [filter]
sysdig is a tool for system troubleshooting, analysis and exploration. It can be used to capture, filter and decode system calls and other OS events.
sysdig can be both used to inspect live systems, or to generate trace files that can be analyzed at a later stage.
sysdig includes a powerul filtering language, has customizable output, and can be extended through Lua scripts, called chisels.
Output format
By default, sysdig prints the information for each captured event on a single line, with the following format:
<evt.num> <evt.time> <evt.cpu> <proc.name> <thread.tid> <evt.dir> <evt.type> <evt.args>
where:
evt.num is the incremental event number
evt.time is the event timestamp
evt.cpu is the CPU number where the event was captured
proc.name is the name of the process that generated the event
thread.tid id the TID that generated the event, which corresponds to the PID for single thread processes
evt.dir is the event direction, > for enter events and < for exit events
evt.type is the name of the event, e.g. \[aq]open\[aq] or \[aq]read\[aq]
evt.args is the list of event arguments.
The output format can be customized with the -p switch, using any of the fields listed by \[aq]sysdig -l\[aq].
Trace Files
A trace file can be created using the -w switch:
$ sysdig -w trace.scap
The -s switch can be used to specify how many bytes of each data buffer should be saved to disk. And filters can be
used to save only certain events to disk:
$ sysdig -s 2000 -w trace.scap proc.name=cat
Trace files can be read this using the -r switch:
$ sysdig -r trace.scap
Filtering
sysdig filters are specified at the end of the command line. The simplest filter is a basic field-value check:
$ sysdig proc.name=cat
The list of available fields can be obtained with \[aq]sysdig -l\[aq].
Filter expressions can use one of these comparison operators: =, !=, <, <=, >, >= and contains. e.g.
$ sysdig fd.name contains /etc
Multiple checks can be combined through brakets and the following boolean operators: and, or, not. e.g.
$ sysdig "not (fd.name contains /proc or fd.name contains /dev)"
Chisels
sysdig\[aq]s chisels are little scripts that analyze the sysdig event stream to perform useful actions.
To get the list of available chisels, type
$ sysdig -cl
To get details about a specific chisel, type
$ sysdig -i spy_ip
To run one of the chisels, you use the -c flag, e.g.
$ sysdig -c topfiles_bytes
If a chisel needs arguments, you specify them after the chisel name:
$ sysdig -c spy_ip 192.168.1.157
If a chisel has more than one argument, specify them after the chisel name, enclosed in quotes:
$ sysdig -c chisel_name "arg1 arg2 arg3"
Chisels can be combined with filters:
$ sysdig -c topfiles_bytes "not fd.name contains /dev"
-A, --print-ascii
Only print the text portion of data buffers, and echo end-of-lines. This is useful to only display human-readable data.
-b, --print-base64
Print data buffers in base64. This is useful for encoding binary data that needs to be used over media designed to handle textual data (i.e., terminal or json).
-c chiselname chiselargs, --chisel=chiselname chiselargs
run the specified chisel. If the chisel require arguments, they must be specified in the command line after the name.
-cl, --list-chisels
lists the available chisels. Looks for chisels in ., ./chisels, ~/.chisels and /usr/share/sysdig/chisels.
-d, --displayflt
Make the given filter a display one. Setting this option causes the events to be filtered after being parsed by the state system. Events are normally filtered before being analyzed, which is more efficient, but can cause state (e.g. FD names) to be lost.
-D, --debug
Capture events about sysdig itself
-F, --fatfile
Enable fatfile mode. When writing in fatfile mode, the output file will contain events that will be invisible when reading the file, but that are necessary to fully reconstruct the state. Fatfile mode is useful when saving events to disk with an aggressive filter. The filter could drop events that would the state to be updated (e.g. clone() or open()). With fatfile mode, those events are still saved to file, but \[aq]hidden\[aq] so that they won\[aq]t appear when reading the file. Be aware that using this flag might generate substantially bigger traces files.
-h, --help
Print this page
-j, --json
Emit output as json, data buffer encoding will depend from the print format selected.
-i chiselname, --chisel-info=chiselname
Get a longer description and the arguments associated with a chisel found in the -cl option list.
-L, --list-events
List the events that the engine supports
-l, --list
List the fields that can be used for filtering and output formatting. Use -lv to get additional information for each field.
-n num, --numevents=num
Stop capturing after num events
-P, --progress
Print progress on stderr while processing trace files.
-p outputformat, --print=outputformat
Specify the format to be used when printing the events. See the examples section below for more info.
-q, --quiet
Don\[aq]t print events on the screen. Useful when dumping to disk.
-r readfile, --read=readfile
Read the events from readfile.
-S, --summary
print the event summary (i.e. the list of the top events) when the capture ends.
-s len, --snaplen=len
Capture the first len bytes of each I/O buffer. By default, the first 80 bytes are captured. Use this option with caution, it can generate huge trace files.
-t timetype, --timetype=timetype
Change the way event time is displayed. Accepted values are h for human-readable string, a for absolute timestamp from epoch, r for relative time from the beginning of the capture, d for delta between event enter and exit, and D for delta from the previous event.
-v, --verbose
Verbose output.
--version
Print version number.
-w writefile, --write=writefile
Write the captured events to writefile.
-x, --print-hex
Print data buffers in hex.
-X, --print-hex-ascii
Print data buffers in hex and ASCII.
-z, --compress
Used with -w, enables compression for tracefiles.
Capture all the events from the live system and print them to screen
$ sysdig
Capture all the events from the live system and save them to disk
$ sysdig -w dumpfile.scap
Read events from a file and print them to screen
$ sysdig -r dumpfile.scap
Print all the open system calls invoked by cat
$ sysdig proc.name=cat and evt.type=open
Print the name of the files opened by cat
$ sysdig -p"%evt.arg.name" proc.name=cat and evt.type=open
List the available chisels
$ sysdig -cl
Use the spy_ip chisel to look at the data exchanged with 192.168.1.157:
$ sysdig -c spy_ip 192.168.1.157
/usr/share/sysdig/chisels
The global chisels directory.
~/.chisels
The personal chisels directory.
sysdig and its chisels are designed to be used with LuaJIT in Lua 5.1 mode. While it is possible to use sysdig with LuaJIT in Lua 5.2 mode or regular Lua, some chisels may not work as expected.
Draios inc. <[email protected]>
strace(8), tcpdump(8), lsof(8)