$filterdir/nbdkit-xz-filter.so
The filter.
Use "nbdkit --dump-config" to find the location of $filterdir.
nbdkit-xz-filter1
Nbdkit xz filter
Usage
nbdkit --filter=xz
file
FILENAME.xz
nbdkit --filter=xz
curl
https://example.com/FILENAME.xz
Overview
"nbdkit-xz-filter" is a filter for nbdkit(1) which uncompresses the underlying plugin on the fly. The filter only supports read-only connections.
Getting best random access performance from xz
xz(1) files are split into streams and blocks. Most xz files contain only one stream which contains one or more blocks. You can find out how many streams and blocks are in an xz file by doing:
$ xz --list winxp.img.xzStrms Blocks Compressed Uncompressed Ratio Check Filename1 1 2,100.0 MiB 6,144.0 MiB 0.342 CRC64 winxp.img.xz↑↑↑ ↑↑↑streams blocks
xz files are seekable on block boundaries only. Seeking is done by seeking directly to the lower block boundary, then uncompressing data until the precise byte is reached. As uncompressing xz data is slow, to get best random access performance, you must prepare your xz files with many small blocks.
Use the --block-size option with a small-ish block size. For example this is the same image compressed with a 16 MB block size:
$ xz --best --block-size=16777216 winxp.img─────────────────────$ xz --list winxp.img.xzStrms Blocks Compressed Uncompressed Ratio Check Filename1 384 2,120.1 MiB 6,144.0 MiB 0.345 CRC64 winxp.img.xz↑↑↑blocks
This file can be accessed randomly. At most 16 MB will have to be uncompressed to seek to any byte.
As you would expect, xz cannot compress as efficiently when using a smaller block size. The space penalty in the above example is < 1% of the compressed file size.
Parameters
xz-max-block=SIZE-
The maximum block size that the filter will read. The filter will refuse to read xz files that contain any block larger than this size.
See the discussion above about creating xz files with small block sizes in order to reduce memory usage and increase performance.
This parameter is optional. If not specified it defaults to 512M.
xz-max-depth=N-
Maximum number of blocks stored in the LRU block cache.
This parameter is optional. If not specified it defaults to 8.
The filter may allocate up to maximum block size in file × maxdepth bytes of memory per connection.
Files
Version
"nbdkit-xz-filter" first appeared in nbdkit 1.10. It is derived from an earlier plugin that first appeared in nbdkit 1.0.
Releases
4 releases since Debian 10 (Buster); the text changed 3 times.