ddb4

Overview

The ddb debugger provides a means for debugging the kernel, and analysing the kernel after a system crash ("panic"), with a gdb(1)-like syntax.

ddb is invoked upon a kernel panic when the sysctl(8) ddb.panic is set to 1. It may be invoked from the console when the sysctl ddb.console is set to 1, using any of the following methods:

• Using the key sequence Ctrl-Alt-Esc.

• Sending a BREAK when using a serial console.

• Writing to the sysctl ddb.trigger.

• For i386 and amd64 architectures, using the key sequence Ctrl-Alt-Delete when the sysctl machdep.kbdreset is set to 2.

ddb prompts for commands on the console with:

ddb>

The general syntax of a ddb command is:

To save typing, ddb makes use of a context inferred from previous commands. In this context, the current location is called dot. The examine, search, show struct, and write commands update dot to be that of the last address examined or the last location modified, and have intuitive effects on next and prev. All the other commands do not change dot, and set next to be the same. (See VARIABLES.)

An expression can be used in place of address (see EXPRESSIONS). Omitting address in a command uses the last value of dot. A missing count is taken to be 1 for printing commands or for stack traces. Entering a blank line causes the last command to be repeated using next in place of address, a count of 1, and no modifiers.

ddb has a feature like more(1) for the output. If the number of lines output in response to one command exceeds the number set in the $lines variable, it displays the message ‘--db_more--’ and waits for a response.

The valid responses are:

⟨space⟩

One more page.

⟨return⟩

One more line.

q

Abort the current command, and return to the command input mode.

The following command line editing keys are provided:

^b

back one character

^f

forward one character

^a

beginning of line

^e

end of line

^w

erase word back

^h | ⟨del⟩

erase previous character

^d

erase next character

^k

delete to end of line

^u

delete line

^p

previous in command history

^n

next in command history

^r

redraw line

^t

exchange the two characters to the left of the cursor

Commands

The following commands may be typed at the ‘ddb>’ prompt. Some commands consist of more than one word, and if only the first word or words are entered, the possible alternatives to complete the command are displayed and no other action is performed.

help

List the available commands.

[e] x[amine] [/bhlqaAxzodurcsmiI][addr] [,count] Display the contents at address addr according to the formats in the modifier. If no format is specified, the last formats specified for this command are used.

The format characters are:

/b

look at by bytes (8 bits)

/h

look at by half words (16 bits)

/l

look at by long words (32 bits) (default)

/q

look at by long longs (64 bits) (only available on 64-bit platforms)

/a

print the location being displayed

/A

print the location with a line number if possible

/x

display in unsigned hex

/z

display in signed hex

/o

display in unsigned octal

/d

display in signed decimal

/u

display in unsigned decimal

/r

display in current radix, signed

/c

display low 8 bits as a character. Non-printing characters are displayed as an octal escape code (e.g., '\000').

/s

display the null-terminated string at the location. Non-printing characters are displayed as octal escapes.

/m

display in unsigned hex with character dump at the end of each line. The location is also displayed in hex at the beginning of each line.

/i

display as an instruction

/I

display as an alternate format instruction depending on the machine:

alpha

Print affected register contents for every instruction.

amd64,

i386

Do not skip padding to the next long word boundary for unconditional jumps.

m88k

Decode instructions for the opposite CPU model (e.g. m88110 when running on an m88100 processor).

The value of next is set to the addr plus the size of the data examined.

p[rint] [/axzodurc] [addr] Print addr according to the modifier character. The valid modifiers are a subset of those from the examine command, and act as described there. If no modifier is specified, the last one specified in a previous use of print is used.

For example,

print/x $eax

will print something like this:

xxxxxx

pp[rint] [addr] Pretty-print addr using CTF debug symbols included in the kernel binary image. The CTF section is normally added by running ctfstrip(1) as part of building a new kernel.

w[rite] [/bhl] [addr] expr [expr ...] Write the value of each expr expression at succeeding locations start at addr. The write unit size can be specified using one of the modifiers:

/b

byte (8 bits)

/h

half word (16 bits)

/l

long word (32 bits) (default)

The value of next is set to addr plus the size of values written.

Warning: since there is no delimiter between expressions, the command may not parse as you expect. It is best to enclose each expression in parentheses.

set

$name [=] expr Set the named variable or register with the value of expr. For valid variable names, see VARIABLES.

boot how

Reboot the machine depending on how:

boot sync

Sync disks and reboot.

boot crash

Dump core and reboot.

boot dump

Sync disks, dump core and reboot.

boot halt

Just halt.

boot reboot

Just reboot.

boot poweroff

Power down the machine whenever possible; if it fails, just halt.

break[addr] [,count] Set a break point at addr. If count is supplied, ddb allows the breakpoint to be silently hit (count - 1) times before stopping at the break point.

d[elete] [addr] Delete the break point set with the break command.

s[tep] [/p] [,count] Single step count times. If the /p modifier is specified, print each instruction at each step. Otherwise, only print the last instruction.

Warning: depending on machine type, it may not be possible to single-step through some low-level code paths. On machines with software-emulated single-stepping (e.g., alpha), stepping through code executed by interrupt handlers will probably do the wrong thing.

call name

(expr [, expr ...] ) Call the function named by name with the argument(s) listed in parentheses. Parentheses may be omitted if the function takes no arguments. The number of arguments is currently limited to 10.

c[ontinue] [/c] Continue execution until a breakpoint or watchpoint. If the /c modifier is given, instructions are counted while executing.

Warning: when counting with /c, ddb is really silently single-stepping. This means that single-stepping on low-level code may cause strange behavior.

watch addr [,size] Set a watchpoint for the region starting at addr. Execution stops and control returns to ddb when an attempt is made to modify a watched region. The size argument defaults to 4.

If you specify a wrong space address, the request is rejected with an error message.

Warning: attempts to watch wired kernel memory may cause an unrecoverable error on some systems (e.g., i386).

dwatch addr

Delete the watchpoint at address addr that was previously set with a watch command.

hangman [/s[0-9]] This is a tiny and handy tool for random kernel hangs analysis, of which its depth is controlled by the optional argument of the default value of five. It uses some sophisticated heuristics to spot the global symbol that caused the hang. Since the discovering algorithm is a probabilistic one, you may spend substantial time to figure the exact symbol name. This smart thing requires a little of your attention, the input it accepts is mostly of the same format as that of the famous hangman(6) game, to which it, apparently, is obliged by the name. Hint: the nm(1) utility might help.

until [/p]

Stop at the next "call" or "return" instruction. If the /p modifier is specified, ddb prints the call nesting depth and the cumulative instruction count at each call or return. Otherwise, it stays silent until the matching return is hit.

match [/p]

Stop at the next matching return instruction. If the /p modifier is specified, ddb prints the call nesting depth and the cumulative instruction count at each call or return. Otherwise, it remains mostly quiet.

next [/p]

The next command is a synonym for match.

kill pid

Send an uncatchable SIGABRT signal to the process specified by the pid argument.

stop pid

Send a SIGSTOP signal to the process specified by the pid argument.

trace [/tu][frameaddr] [,count] Show the stack trace. The /t modifier interprets the frameaddr argument as the TID of a process and shows the stack trace of that process. frameaddr is subject to the radix; use the 0t prefix to enter a decimal TID. The /t modifier is not supported on all platforms. The /u modifier shows the stack trace of user space; if omitted, the kernel stack is traced instead. The count argument is the limit on the number of frames to be followed. If count is omitted, all frames are printed.

Warning: user space stack trace is valid only if the machine dependent code supports it.

search [/bhl] [addr] value [mask] [,count] Search memory for a value beginning at addr. This command might fail in interesting ways if it doesn't find the searched-for value. This is because ddb doesn't always recover from touching bad memory. The optional count argument limits the search. The modifiers are the same as those of the write command.

The next address is set to the address where value is found, or just after where the search area finishes.

reboot

Shortcut for boot reboot

show what

Displays various things, depending on what:

show bcstats

Prints the buffer cache statistics.

show breaks

Prints a list of all breakpoints that have been set with the break command.

show buf addr

Prints the struct buf at addr.

show extents

Prints a detailed list of all extents.

show locks [addr]

Prints the list of locks held by a thread. If an optional address is not specified, curproc is assumed. The option WITNESS is required for this command to be available.

show malloc [addr]

Prints malloc debugging information if available. If an optional address is specified, only information about that address is printed.

show map [/f] addr Prints the vm_map at addr. If the /f modifier is specified, the complete map is printed.

show mbuf [/cp] addr Prints the struct mbuf header at addr. Depending on the mbuf flags struct pkthdr and struct m_ext are printed as well. If the /c modifier is specified, print the mbuf chain linked with the m_next pointer. /p does the same using m_nextpkt. Both can be combined.

show mount [/f] addr Prints the struct mount at addr. If the /f modifier is specified, prints out all vnodes (see also show vnode) and also all bufs (see also show buf) on all those vnodes.

show nfsnode [/f] addr Prints the struct nfsnode at addr. If the /f modifier is specified, prints out additional information as well.

show nfsreq [/f] addr Prints the struct nfsreq at addr. If the /f modifier is specified, prints out additional information as well.

show object [/f] addr Prints the vm_object at addr. If the /f modifier is specified, the complete object is printed.

show page [/f] addr Prints the vm_page at addr. If the /f modifier is specified, the complete page is printed.

show panic

Prints the panic string.

show pool [/p] addr Prints the pool at addr. Valid modifiers:

/p

Print the pagelist for this pool.

show proc [/t] [addr] Prints the struct proc at addr. If an optional address is not specified, curproc is assumed. The /t modifier interprets the addr argument as the TID of a proc.

show registers [/u]

Display the register set. If the /u modifier is specified, it displays user registers (or the currently saved registers) instead of the kernel's. Note: The /u modifier is not supported on every machine, in which case incorrect information may be displayed.

show route addr

Prints the struct rtentry at addr.

show socket addr

Prints the struct socket at addr. If the socket is spliced, the struct sosplice associated with the socket is printed as well.

show struct name [addr] Prints the content of the memory at addr as a struct name. Nested structures and bit fields are not printed. Character arrays are printed as bytes.

show swap

Prints a detailed list of all swaps.

show tdb [/f] addr Prints the struct tdb at addr. If the /f modifier is specified, prints out all fields of this IPsec SA.

show uvmexp

Displays a selection of uvm counters and statistics.

show vnode [/f] addr Prints the struct vnode at addr. If the /f modifier is specified, prints out all bufs (see also show buf) currently attached to this vnode.

show watches

Displays all watchpoints set with the watch command.

show witness [/b]

Prints the current order list. If the /b modifier is specified, the list of found lock order violations is printed instead. The option WITNESS is required for this command to be available.

show all procs [/anow]

Display information on all processes.

/n

(Default) Show process information in a ps(1)-like format. Information printed includes process ID, thread ID, parent process ID, UID, process status, process flags, process wait channel message and process command name.

/a

Shows the kernel virtual addresses of each process' proc structure, u-area, and vmspace structure. The vmspace address is also the address of the process' vm_map structure and can be used in the show map command.

/o

Shows non-idle threads that were on CPU when ddb was entered. Information printed includes thread ID, process ID, UID, process flags, thread flags, current CPU, and command name.

/w

Shows each thread's ID, command, process group, wait channel address, and wait channel message.

show all bufs [/f]

Display information about all buffers in the system.

/f

For each buffer, print a more detailed output. See the show buf command for more information.

show all callout

Display the contents of the callout table.

show all pools [/a]

Display information about all system pools in a format similar to vmstat(8).

/a

Displays “interesting” address information.

show all locks

Prints the list of locks held by all threads in the system. The option WITNESS is required for this command to be available.

show all mounts [/f]

Display information on all mounted filesystems.

/f

For each filesystem, list all its struct vnode addresses. These addresses can be used in the show vnode command.

show all nfsnodes [/f]

Display information about all nfsnodes in the system.

/f

For each nfsnode, print a more detailed output. See the show nfsnode command for more information.

show all nfsreqs [/f]

Display information for all outstanding NFS requests.

/f

For each NFS requests, print a more detailed output. See the show nfsreq command for more information.

show all routes [/iI] [rtableid] [,count] Show internet routing tables. Default for rtableid is 0 and count is 1.

/i

Restrict to AF_INET.

/I

Restrict to AF_INET6.

show all tdbs [/f]

Display information about all IPsec SAs in the system.

/f

For each tdb, print a more detailed output. See the show tdb command for more information.

show all vnodes [/f]

Display information about all vnodes in the system.

/f

For each vnode, print a more detailed output. See the show vnode command for more information.

callout

A synonym for the show all callout command.

ps [/anow]

A synonym for show all procs.

mac[hine] subcommand [arg ...] Perform a platform-specific command.

The following commands are supported by multiprocessor kernels on these platforms: amd64, i386, macppc, mips64, and sparc64.

cpuinfo

Display the state of each CPU.

ddbcpu N

Stop the current CPU and start handling ddb on the selected CPU.

startcpu [N]

Resume normal processing on the selected CPU, or all CPUs if none is specified.

stopcpu [N]

Stop normal processing on the selected CPU, or all CPUs (except the one handling ddb) if none is specified.

Other platform-specific commands:

arm:

frame addr

Display the trapframe at addr.

i386:

sysregs

Display the contents of the privileged registers: IDTR, GDTR, LDTR, TR, CR0, CR2, CR3, and CR4.

m88k:

ddbcpu N

Stop the current CPU and start handling ddb on the selected CPU.

frame addr

Display the trapframe at addr.

regs

Display the registers from when ddb was entered.

searchframe [addr]

Search for and display stack exception frames, starting from addr if given, else the address in register r31, and stopping at the next 8k boundary.

where

Display where the current CPU was stopped.

mips64:

tlb [/p asid] [/c] [tlb]

trap ??

sh:

cache [addr]

Display the cache, starting from addr, defaulting to 0.

frame

Display the switch and trap frames.

tlb

Display the TLB.

sparc64:

ctx

Display the context addresses for all threads.

Variables

ddb denotes registers and variables by $name. Register names can be found with the show registers command.

Built-in debugger variables currently supported are:

$radix

Input and output radix.

$maxoff

Addresses are printed as symbol+offset unless offset is greater than $maxoff.

$maxwidth

The width of the displayed lines.

$lines

The number of lines to page. This is used by the “more” feature.

$tabstops

Tab stop width.

$log

Controls whether the output of ddb will also appear in the system message buffer.

These variables can also be controlled outside ddb via the ‘ddb’ sysctl(8) hierarchy.

Expressions

Almost all expression operators in C are supported except for ‘~’, ‘^’, and unary ‘&’. Special rules for expressions in ddb are:

identifier

The name of a symbol. It is translated to the address (or value) of the symbol. ‘.’ and ‘:’ can be used in the identifier. The following can be accepted as an identifier, if supported by an object format dependent routine:

[filename:]func [:linenumber]

[filename:] variable

filename [:linenumber]

The symbol may be prefixed with ‘symboltablename::’ (e.g., ‘emulator::mach_msg_trap’) to specify other than kernel symbols.

number

The radix is determined by the first two letters: ‘0x’: hex, ‘0o’: octal, ‘0t’: decimal, otherwise, the value of $radix is used.

.

dot: the current address.

+

next: the next address.

..

The address of the start of the last line examined. Unlike dot or next, this is only changed by the examine or write command.

'

The last address explicitly specified.

$variable

The value of a register or variable.

expr # expr

A binary operator which rounds up the left hand side to the next multiple of right hand side.

*expr

Indirection. It may be followed by a ':' as described above with identifier.