nndocs:infiniband
Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| nndocs:infiniband [2026/08/08 13:45] – update slightly; add links for hardware naptastic | nndocs:infiniband [2026/08/09 06:35] (current) – Move GUIDs to their pages naptastic | ||
|---|---|---|---|
| Line 102: | Line 102: | ||
| If you unset the high bit on the partition number (0x3128 instead of 0xb128) Linux will set the high bit before joining the partition. If OpenSM' | If you unset the high bit on the partition number (0x3128 instead of 0xb128) Linux will set the high bit before joining the partition. If OpenSM' | ||
| - | |||
| - | It's worth finding out if Netplan can manage IB child interfaces. | ||
| ====Connected vs. Datagram==== | ====Connected vs. Datagram==== | ||
| Line 128: | Line 126: | ||
| =====SR-IOV===== | =====SR-IOV===== | ||
| - | |||
| - | ConnectX-3 cards (mlx4*) handle SR-IOV completely differently. These instructions are for the Connect-IB and ConnectX-4 cards (mlx5*). | ||
| ====Hardware Settings==== | ====Hardware Settings==== | ||
| Line 140: | Line 136: | ||
| SRIOV_EN | SRIOV_EN | ||
| - | FPP_EN (Function Per Port ENable) controls whether the card appears as two PCI devices, or as a single device with two ports. Under mlx4, every VF on a dual-port HCA has both ports, and NUM_OF_VFs is how many dual-port devices to create. Under mlx5, each port gets its own pool of VFs and NUM_OF_VFs is per-port. | + | FPP_EN (Function Per Port ENable) controls whether the card appears as two PCI devices, or as a single device with two ports. Under mlx5, each port gets its own pool of VFs and NUM_OF_VFs is per-port. |
| I haven' | I haven' | ||
| - | |||
| - | To make VFs exist, put a number <= NUM_OF_VFS into sriov_numvfs for that device. Before doing so, I recommend turning off VF probing. Otherwise the VFs will all make IPoIB interfaces, which probably isn't what you want. This setting is per PF. | ||
| - | |||
| - | I'm still checking if there' | ||
| - | |||
| - | # echo 0 > / | ||
| - | # echo 0 > / | ||
| - | |||
| - | If it works, there will be new PCI devices as well as VFs listed under `ip link`: | ||
| - | |||
| - | # echo 7 > / | ||
| - | # lspci | grep nfi | ||
| - | 06:00.0 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB] | ||
| - | 06:00.1 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB] | ||
| - | 06:00.2 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] | ||
| - | 06:00.3 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] | ||
| - | 06:00.4 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] | ||
| - | 06:00.5 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] | ||
| - | 06:00.6 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] | ||
| - | 06:00.7 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] | ||
| - | 06:01.0 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] | ||
| - | |||
| - | Warning: The output from ip link is very wide. | ||
| - | |||
| - | # ip link | ||
| - | 1: lo: < | ||
| - | link/ | ||
| - | 2: eth0: < | ||
| - | link/ether 3c: | ||
| - | 3: eth1: < | ||
| - | link/ether 3c: | ||
| - | 4: ib0: < | ||
| - | link/ | ||
| - | vf 0 | ||
| - | vf 1 | ||
| - | vf 2 | ||
| - | vf 3 | ||
| - | vf 4 | ||
| - | vf 5 | ||
| - | vf 6 | ||
| - | 5: ib1: < | ||
| - | link/ | ||
| - | |||
| - | ====VF Configuration==== | ||
| - | The official documentation covers a sysfs interface for configuring VF properties. That interface hasn't existed for years. Before using a VF, you must set node_guid, port_guid, and state using ip link. Make port_guid == node_guid == unique. (I use the base port guid + VF + 1.) | ||
| - | |||
| - | GUIDs need to be set before attaching a VF to a VM. It should be possible to change state (simulating unplugging the cable) while a VM is using a VF but I haven' | ||
| - | |||
| - | I keep the whole configuration in / | ||
| =====Upper-Layer Protocols (ULPs)===== | =====Upper-Layer Protocols (ULPs)===== | ||
| Line 250: | Line 197: | ||
| I also want to throw audio frames around with "no latency added" | I also want to throw audio frames around with "no latency added" | ||
| - | =====GUIDs===== | + | ====GUIDs==== |
| - | ====Installed==== | + | GUIDs and MAC addresses for Mellanox cards are listed |
| - | * b8cef603004ebc86 - southpark ConnectX-4 FCAT | + | |
| - | * 0002c90300b37f10 - shark ConnectX-3 FCBT with no date on the label | + | |
| - | * 5849560e53660101 - duckling Connect-IB | + | |
| - | ====Not Installed==== | + | |
| - | * 5849560e53b70b01 - Connect-IB | + | |
| - | * 5849560e59150301 - Connect-IB | + | |
| - | * 7cfe900300a0a080 - Connect-IB | + | |
| - | * (there | + | |
| - | * f4521403002c18b0 - uninstalled ConnectX-3 dated 2014-01-29 | + | |
| * 001175000079b560 - uninstalled qib | * 001175000079b560 - uninstalled qib | ||
| * 001175000079b856 - uninstalled qib | * 001175000079b856 - uninstalled qib | ||
nndocs/infiniband.1786196745.txt.gz · Last modified: 2026/08/08 13:45 by naptastic