Naptastic

User Tools

Site Tools

Trace:
nndocs:infiniband:connectx-4

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

nndocs:infiniband:connectx-4 [2026/08/08 19:45] – created naptasticnndocs:infiniband:connectx-4 [2026/08/08 21:04] (current) – [SR-IOV] All the info I currently have naptastic
Line 139: Line 139:
  
 ====SR-IOV==== ====SR-IOV====
-The ConnectX-4 is definitely where SR-IOV is most useful. Right now, I know it works, but that's about it. Since it's configured IB/ETH and supports switchdev mode, it should be a lot better for use in a hypervisor.+(For info about Infiniband SR-IOV, also read the [[nndocs::infiniband::Connect-IB]] page.)
  
-I wonder if it works with virt-manager. That would be so awesome.+Create virtual functions: 
 + 
 +  [root]@[southpark][13:49:01][/sys/bus/pci/devices/0000:06:00.1]# echo 0 > sriov_drivers_autoprobe  
 +  [root]@[southpark][13:49:26][/sys/bus/pci/devices/0000:06:00.1]# echo 8 > sriov_numvfs  
 +   
 +  [root]@[southpark][13:49:44][/sys/bus/pci/devices/0000:06:00.1]# ip link 
 +  (...snip...) 
 +  8: mlx1: <BROADCAST,MULTICAST> mtu 4042 qdisc mq master green state DOWN mode DEFAULT group default qlen 1000 
 +      link/ether b8:ce:f6:4e:bc:87 brd ff:ff:ff:ff:ff:ff 
 +      vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off 
 +      vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off 
 +      vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off 
 +      vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off 
 +      vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off 
 +      vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off 
 +      vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off 
 +      vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off 
 +  (...snip...) 
 + 
 +This adapter helpfully creates new PCI devices for the VFs instead of adding more functions to device 0. 
 + 
 +  [root]@[southpark][13:49:47][/sys/bus/pci/devices/0000:06:00.1]# lspci | grep Mel 
 +  06:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4] 
 +  06:00.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4] 
 +  06:01.2 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function] 
 +  06:01.3 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function] 
 +  06:01.4 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function] 
 +  06:01.5 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function] 
 +  06:01.6 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function] 
 +  06:01.7 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function] 
 +  06:02.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function] 
 +  06:02.1 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function] 
 + 
 +Configure a VF; this one is for the Factorio server: 
 + 
 +  [root]@[southpark][14:26:26][~]# ip link set mlx1 vf 0 mac 00:02:c8:00:00:38 vlan 3 
 +  [root]@[southpark][14:26:26][~]# ip link 
 +        vf 0     link/ether 00:02:c8:00:00:38 brd ff:ff:ff:ff:ff:ff, vlan 3, spoof checking off, link-state auto, trust off, query_rss off 
 +        vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off 
 + 
 +(Notice vlan wasn't there before, but it's there now.) 
 + 
 +===For a virtual machine=== 
 + 
 +The SR-IOV network pool thing in virt-manager still doesn't work. It says the PF isn't configured. I haven't dug into it at all. You have to attach the PCIe host device for the VF you want to use. Here's the working XML that virt-manager generated for factorio: 
 + 
 +  <hostdev mode="subsystem" type="pci" managed="yes"> 
 +    <driver name="vfio"/> 
 +    <source> 
 +      <address domain="0x0000" bus="0x06" slot="0x01" function="0x2"/> 
 +    </source> 
 +    <alias name="hostdev0"/> 
 +    <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/> 
 +  </hostdev> 
 + 
 +===For use by the host=== 
 + 
 +There's a sysfs interface for managing driver overrides, or you can use driverctl, which is not installed by default. 
 + 
 +Warning: `driverctl list-devices` lists **all** the devices on the PCI bus: the root complex, the ports, the fabric functions, anything that looks like a bridge... on southpark, it's ~2 pages of output. 
 + 
 +Strongly recommend grepping the output for the relevant PCI devices with a start-of-line anchor and the PCI domain: 
 + 
 +  # driverctl list-devices | grep ^0000:06:0 
 +  0000:06:00.0 mlx5_core 
 +  0000:06:00.1 mlx5_core 
 +  0000:06:01.2 vfio-pci [*] 
 +  0000:06:01.3 mlx5_core [*] 
 +  0000:06:01.4 (none) 
 +  0000:06:01.5 (none) 
 +  0000:06:01.6 (none) 
 +  0000:06:01.7 (none) 
 +  0000:06:02.0 (none) 
 +  0000:06:02.1 (none) 
 + 
 +Configure the VF as above: 
 + 
 +  [root]@[southpark][14:26:29][~]# ip link set mlx1 vf 1 mac 00:10:18:f8:02:22 vlan 3 
 +  [root]@[southpark][14:34:24][~]# ip link 
 +  8: mlx1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4042 qdisc mq master green state UP mode DEFAULT group default qlen 1000 
 +      link/ether b8:ce:f6:4e:bc:87 brd ff:ff:ff:ff:ff:ff 
 +      vf 0     link/ether 00:02:c8:00:00:38 brd ff:ff:ff:ff:ff:ff, vlan 3, spoof checking off, link-state auto, trust off, query_rss off 
 +      vf 1     link/ether 00:10:18:f8:02:22 brd ff:ff:ff:ff:ff:ff, vlan 3, spoof checking off, link-state auto, trust off, query_rss off 
 +      vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off 
 + 
 +Bind the driver to mlx5_core: 
 + 
 +  [root]@[southpark][14:36:01][/sys/bus/pci/devices/0000:06:00.1]# driverctl set-override 0000:06:01.3 mlx5_core 
 +   
 +  [root]@[southpark][14:55:00][/sys/bus/pci/devices/0000:06:00.1]## driverctl list-devices | grep ^0000:06:0 
 +  0000:06:00.0 mlx5_core 
 +  0000:06:00.1 mlx5_core 
 +  0000:06:01.2 vfio-pci [*] 
 +  0000:06:01.3 mlx5_core [*] 
 +  (...snip...) 
 + 
 +A new network device will appear in `ip link`. Here I have already connected it to green and turned it on: 
 + 
 +  21: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4042 qdisc mq master green state UP mode DEFAULT group default qlen 1000 
 +    link/ether 00:10:18:f8:02:22 brd ff:ff:ff:ff:ff:ff
  
 ====GUIDs==== ====GUIDs====
 Right now I only have one ConnectX-4: Right now I only have one ConnectX-4:
   * b8cef603004ebc86 - CX456A-FCAT installed in southpark   * b8cef603004ebc86 - CX456A-FCAT installed in southpark
nndocs/infiniband/connectx-4.txt · Last modified: 2026/08/08 21:04 by naptastic