ntb_hw_plx4

Usage

Overview

The ntb_hw_plx driver provides support for the Non-Transparent Bridge (NTB) hardware in PLX PCIe bridge chips, which allow up to two of their PCIe ports to be switched from transparent to non-transparent bridge mode. In this mode bridge looks not as a PCI bridge, but as PCI endpoint device. The driver hides hardware details, exposing memory windows, scratchpads and doorbells of the other side via hardware independent KPI to ntb(4) subsystem.

Each PLX NTB provides up to 2 64-bit or 4 32-bit memory windows to the other system's memory, 6 or 12 scratchpad registers and 16 doorbells to interrupt the other system. If Address Lookup Table (A-LUT) is enabled, BAR2 can be split into several (up to 128) memory windows. In NTB-to-NTB mode one of memory windows (or half of it, if bigger then 1MB) is consumed by the driver itself to access scratchpad and doorbell registers of the other side.

Hardware

The ntb_hw_plx driver supports the following PLX/Avago/Broadcom Non-transparent Bridge controllers:

• PEX 8713

• PEX 8717

• PEX 8725

• PEX 8733

• PEX 8749

Configuration

The basic chip configuration should be done by serial EEPROM or via i2c. It includes enabling NTB on one or both sides (choosing between NTB-to-NTB (back-to-back) and NTB-to-Root Port modes) and configuring BARs sizes.

The recommended mode is NTB-to-NTB mode, since while NTB-to-Root Port is generally supported by the driver, it require PCI hotplug handling on the Root Port, that may be difficult or cause different kinds of problems.

Loader tunables

The following tunables are settable from the loader(8):

hint.ntb_hw.%d.b2b

Being set to 1 (default) tells the driver attached to Virtual Interface of the NTB that it works in NTB-to-NTB (back-to-back) mode, 0 -- NTB-to-Root Port. Driver attached to Link Interface (visible from Root Port side) switches to NTB-to-Root Port mode automatically, but one attached to Virtual Interface can't detect what is on the other side and require external knowledge.

hint.ntb_hw.%d.split

Being set above zero splits BAR2 into 2^x memory windows using Address Lookup Table (A-LUT).