Naptastic

User Tools

Site Tools

Trace:
nndocs:infiniband

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
nndocs:infiniband [2024/12/14 17:59] – update SR-IOV config naptasticnndocs:infiniband [2026/08/09 06:35] (current) – Move GUIDs to their pages naptastic
Line 1: Line 1:
 =====InfiniBand===== =====InfiniBand=====
 Configuration is a real pain. On my Debian hosts, it's all in /etc/rc.local, which is gross. I haven't figured out how to do it on Ubuntu, since Netplan apparently has support for vxlan and IP over IB (IPoIB) now, I think. **We'll get there**. Configuration is a real pain. On my Debian hosts, it's all in /etc/rc.local, which is gross. I haven't figured out how to do it on Ubuntu, since Netplan apparently has support for vxlan and IP over IB (IPoIB) now, I think. **We'll get there**.
 +
 +  * [[nndocs::infiniband::ConnectX-3]]
 +  * [[nndocs::infiniband::Connect-IB]]
 +  * [[nndocs::infiniband::ConnectX-4]]
 +  * [[SX6005]]
 +  * [[SX6036]]
  
 ===The Problem=== ===The Problem===
-All the InfiniBand hardware I have is Mellanox FDR-generation, so, ConnectX-3, Connect-IB, and SX6005 IB-only switches. (Every time I think "the Ethernet version sure would be nice" I remind myself that I'd use InfiniBand mode anyway.)+All the InfiniBand hardware I have is Mellanox FDR-generation.
  
-For hardware support, Mellanox provides MLNX_OFED, an overlay for several distributions. Unfortunately, MLNX_OFED only supports Debian through 11, RHEL through 8, and Ubuntu through 20.04, **OR** ConnectX-4 or newer cards only. The drivers built into Linux still recognize and work with ConnectX-3 and Connect-IB, but the kernel drivers packaged won't even recognize the cards. They just show up as unclaimed PCI devices.+For hardware support, Mellanox provides MLNX_OFED, an overlay for several distributions. Unfortunately, MLNX_OFED only supports Debian through 11, RHEL through 8, and Ubuntu through 20.04, **OR** ConnectX-4 or newer cards only. The drivers built into Linux still recognize and work with ConnectX-3 and Connect-IB, but the kernel drivers in MLNX_OFED don't. They just show up as unclaimed PCI devices.
  
 ^ Version ^ Minimum hardware ^ Debian ^ Ubuntu ^ OpenSM version ^ ^ Version ^ Minimum hardware ^ Debian ^ Ubuntu ^ OpenSM version ^
Line 78: Line 84:
 Here's a block for my ATA over Ethernet experiments. Subject to change. IP addresses are necessary for setting up VXLAN tunnels. Checking if IPv6 tunnels perform differently from IPv4 tunnels is on the to-do list. I suspect they perform better. Needs testing. Here's a block for my ATA over Ethernet experiments. Subject to change. IP addresses are necessary for setting up VXLAN tunnels. Checking if IPv6 tunnels perform differently from IPv4 tunnels is on the to-do list. I suspect they perform better. Needs testing.
  
-  mp1=0x3128, ipoib, rate=12, mtu=5, scope=2, defmember=full:+  storage=0xb128, ipoib, rate=12, mtu=5, scope=2, defmember=full:
       mgid=ff12:401b::ffff:ffff   # IPv4 Broadcast address       mgid=ff12:401b::ffff:ffff   # IPv4 Broadcast address
       mgid=ff12:401b::1           # IPv4 All Hosts group       mgid=ff12:401b::1           # IPv4 All Hosts group
Line 85: Line 91:
 ====Partitions: Host configuration==== ====Partitions: Host configuration====
 There's no functional netlink interface for creating child interfaces. You must use the sysfs interface. There's no functional netlink interface for creating child interfaces. You must use the sysfs interface.
-  # echo 0xb129 > /sys/class/net/ib0/create_child +  # echo 0xb128 > /sys/class/net/ib0/create_child
- +
-The sysfs interface for deleting child interfaces doesn't work (for me at least). You must use the netlink interface. +
-  # ip link del ib0.b129+
  
 Resist the temptation to rename the interface to something descriptive. **It's already self-descriptive**. Creative naming is for VXLAN tunnels and bridges, e.g.: Resist the temptation to rename the interface to something descriptive. **It's already self-descriptive**. Creative naming is for VXLAN tunnels and bridges, e.g.:
  
-  # ip link add vx129 type vxlan id 129 local 172.20.129.9 group 225.172.20.129 +  # ip link add vx128 type vxlan id 128 local 172.20.128.13 group 225.172.20.128 
-  # ip link set master aoe2 dev vx129+  # ip link set master aoe1 dev vx128
  
-If you unset the high bit on the partition number (0x3129 instead of 0xb129) Linux will set the high bit before joining the partition. If OpenSM's configuration has that partition's membership set for "partial" or "both", the Linux host will not be able to connect to everything on that subnet, or possibly //anything// on that subnet, regardless of which value you use.+The sysfs interface for deleting child interfaces doesn't work (for me at least). You must use the netlink interface. 
 +  # ip link del ib0.b128
  
-It's worth finding out if Netplan can manage IB child interfaces.+If you unset the high bit on the partition number (0x3128 instead of 0xb128) Linux will set the high bit before joining the partition. If OpenSM's configuration has that partition's membership set for "partial" or "both", the Linux host will not be able to connect to everything on that subnet, or possibly //anything// on that subnet, regardless of which value you use.
  
 ====Connected vs. Datagram==== ====Connected vs. Datagram====
Line 122: Line 126:
  
 =====SR-IOV===== =====SR-IOV=====
 +
 ====Hardware Settings==== ====Hardware Settings====
 The BIOS needs to have SR-IOV, ARI, and ACS support enabled. The BIOS needs to have SR-IOV, ARI, and ACS support enabled.
Line 131: Line 136:
     SRIOV_EN                                    True(1)     SRIOV_EN                                    True(1)
  
-FPP_EN (Function Per Port ENable) controls whether the card appears as two PCI devices, or as a single device with two ports. Under mlx4, every VF on a dual-port HCA has both ports, and NUM_OF_VFs is how many dual-port devices to create. Under mlx5, each port gets its own pool of VFs and NUM_OF_VFs is per-port.+FPP_EN (Function Per Port ENable) controls whether the card appears as two PCI devices, or as a single device with two ports. Under mlx5, each port gets its own pool of VFs and NUM_OF_VFs is per-port.
  
-I haven't tried large numbers of VFs. The hardware upper limit is 63 for Connect-IB and 127 for ConnectX-3. Any number of system components could impose lower limits. For example, my consumer boards that are SR-IOV capable can only have VFs on port 1, not on port 2; the EPYC server system can create VFs on both ports. I don't expect to need so many guests with IOV networking anyway... +I haven't tried large numbers of VFs. The hardware upper limit is 63 for Connect-IB and 127 for ConnectX-3. Any number of system components could impose lower limits. I don't expect to need so many guests with IOV networking anyway...
- +
-To make VFs exist, put a number <= NUM_OF_VFS into sriov_numvfs for that device. Before doing so, I recommend turning off VF probing. Otherwise the VFs will all make IPoIB interfaces, which probably isn't what you want. This setting is per PF. +
- +
-I'm still checking if there's a way to configure the driver so this becomes the default setting. +
- +
-  # echo 0 > /sys/class/infiniband/ibp13s0f0/device/sriov_drivers_autoprobe +
-  # echo 0 > /sys/class/infiniband/ibp13s0f1/device/sriov_drivers_autoprobe +
- +
-If it works, there will be new PCI devices as well as VFs listed under `ip link`: +
- +
-  # echo 7 > /sys/class/infiniband/ibp6s0f0/device/sriov_numvfs # no output on success; check dmesg for interesting but probably useless messages. +
-  # lspci | grep nfi +
-  06:00.0 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB] +
-  06:00.1 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB] +
-  06:00.2 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
-  06:00.3 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
-  06:00.4 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
-  06:00.5 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
-  06:00.6 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
-  06:00.7 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
-  06:01.0 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
- +
-Warning: The output from ip link is very wide. +
- +
-  # ip link +
-  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 +
-      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 +
-  2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 +
-      link/ether 3c:ec:ef:6d:10:62 brd ff:ff:ff:ff:ff:ff +
-  3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 +
-      link/ether 3c:ec:ef:6d:10:63 brd ff:ff:ff:ff:ff:ff +
-  4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 qdisc fq_codel state UP mode DEFAULT group default qlen 256 +
-      link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff +
-      vf 0     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-      vf 1     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-      vf 2     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-      vf 3     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-      vf 4     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-      vf 5     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-      vf 6     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-  5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc fq_codel state DOWN mode DEFAULT group default qlen 256 +
-      link/infiniband 80:00:00:28:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:09 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff +
- +
-====VF Configuration==== +
-The official documentation covers a sysfs interface for configuring VF properties. That interface hasn't existed for years. Before using a VF, you must set node_guid, port_guid, and state using ip link. Make port_guid == node_guid == unique. (I use the base port guid + VF + 1.) +
- +
-GUIDs need to be set before attaching a VF to a VM. It should be possible to change state (simulating unplugging the cable) while a VM is using a VF but I haven't tested this. +
- +
-This is in /etc/rc.local for southpark: +
-  ip link set dev ib0 vf 0 node_guid 58:49:56:0e:53:b7:0b:02 +
-  ip link set dev ib0 vf 0 port_guid 58:49:56:0e:53:b7:0b:02 +
-  ip link set dev ib0 vf 0 state enable +
-  ip link set dev ib0 vf 1 node_guid 58:49:56:0e:53:b7:0b:03 +
-  ip link set dev ib0 vf 1 port_guid 58:49:56:0e:53:b7:0b:03 +
-  ip link set dev ib0 vf 1 state enable +
-  ip link set dev ib0 vf 2 node_guid 58:49:56:0e:53:b7:0b:04 +
-  ip link set dev ib0 vf 2 port_guid 58:49:56:0e:53:b7:0b:04 +
-  ip link set dev ib0 vf 2 state enable +
-  ip link set dev ib0 vf 3 node_guid 58:49:56:0e:53:b7:0b:05 +
-  ip link set dev ib0 vf 3 port_guid 58:49:56:0e:53:b7:0b:05 +
-  ip link set dev ib0 vf 3 state enable +
-  ip link set dev ib0 vf 4 node_guid 58:49:56:0e:53:b7:0b:06 +
-  ip link set dev ib0 vf 4 port_guid 58:49:56:0e:53:b7:0b:06 +
-  ip link set dev ib0 vf 4 state enable +
-  ip link set dev ib0 vf 5 node_guid 58:49:56:0e:53:b7:0b:07 +
-  ip link set dev ib0 vf 5 port_guid 58:49:56:0e:53:b7:0b:07 +
-  ip link set dev ib0 vf 5 state enable +
-  ip link set dev ib0 vf 6 node_guid 58:49:56:0e:53:b7:0b:08 +
-  ip link set dev ib0 vf 6 port_guid 58:49:56:0e:53:b7:0b:08 +
-  ip link set dev ib0 vf 6 state enable +
- +
-Lazy copy-pasta for [this card not currently installed]: +
-  ip link set dev ib0 vf 0 node_guid 58:49:56:0e:58:5c:03:02 +
-  ip link set dev ib0 vf 0 port_guid 58:49:56:0e:58:5c:03:02 +
-  ip link set dev ib0 vf 0 state enable +
-  ip link set dev ib0 vf 1 node_guid 58:49:56:0e:58:5c:03:03 +
-  ip link set dev ib0 vf 1 port_guid 58:49:56:0e:58:5c:03:03 +
-  ip link set dev ib0 vf 1 state enable +
-  ip link set dev ib0 vf 2 node_guid 58:49:56:0e:58:5c:03:04 +
-  ip link set dev ib0 vf 2 port_guid 58:49:56:0e:58:5c:03:04 +
-  ip link set dev ib0 vf 2 state enable +
-  ip link set dev ib0 vf 3 node_guid 58:49:56:0e:58:5c:03:05 +
-  ip link set dev ib0 vf 3 port_guid 58:49:56:0e:58:5c:03:05 +
-  ip link set dev ib0 vf 3 state enable +
-  ip link set dev ib0 vf 4 node_guid 58:49:56:0e:58:5c:03:06 +
-  ip link set dev ib0 vf 4 port_guid 58:49:56:0e:58:5c:03:06 +
-  ip link set dev ib0 vf 4 state enable +
-  ip link set dev ib0 vf 5 node_guid 58:49:56:0e:58:5c:03:07 +
-  ip link set dev ib0 vf 5 port_guid 58:49:56:0e:58:5c:03:07 +
-  ip link set dev ib0 vf 5 state enable +
-  ip link set dev ib0 vf 6 node_guid 58:49:56:0e:58:5c:03:08 +
-  ip link set dev ib0 vf 6 port_guid 58:49:56:0e:58:5c:03:08 +
-  ip link set dev ib0 vf 6 state enable +
- +
-Lazy (and incomplete) copy-pasta for [also not installed]: +
-  ip link set dev ib0 vf 0 node_guid 58:49:56:0e:59:11:02:02 +
-  ip link set dev ib0 vf 0 port_guid 58:49:56:0e:59:11:02:02 +
-  ip link set dev ib0 vf 0 state enable +
- +
-These should really go on their own page. Or better yet, figure out how to configure them on the host!+
  
 =====Upper-Layer Protocols (ULPs)===== =====Upper-Layer Protocols (ULPs)=====
Line 250: Line 155:
  
 None of these services is so performance-critical that I'll spend time configuring it for VMA, except maybe as a learning exercise. **Later**. None of these services is so performance-critical that I'll spend time configuring it for VMA, except maybe as a learning exercise. **Later**.
 +
 ====Storage==== ====Storage====
   * NFS/RDMA (probably needs a page)   * NFS/RDMA (probably needs a page)
Line 257: Line 163:
   * See [[iSCSI]] for information about enabling and using iSER.   * See [[iSCSI]] for information about enabling and using iSER.
   * See [[NVMe-oF]] for using NVMe over InfiniBand.   * See [[NVMe-oF]] for using NVMe over InfiniBand.
-  * I'm curious about the performance of SRP. I have gotten it to work before but it's not working now. Also not a huge priority.+  * [[SRP]] is fast but the documentation is a joke. 
 ====Networking==== ====Networking====
 ===VXLAN=== ===VXLAN===
-VXLAN is not the only way to get an Ethernet device on Infiniband, but as far as I can tell it's the only decent one. Neither ConnectX-3 nor Connect-IB has VXLAN offload support. Despite this, in connected mode, VXLAN is still really fast.+VXLAN is not the only way to get an Ethernet device on Infiniband, but as far as I can tell it's the only decent one. 
 + 
 +ConnectX-3 Pro, ConnectX-4, and newer Mellanox cards have VXLAN offloading support, but only for Ethernet ports. So VXLAN overlays are always going to be done by the CPU. They're still decently fast.
  
   * VXLAN id can be anything from 0-16777215 inclusive. I make it match the network number.   * VXLAN id can be anything from 0-16777215 inclusive. I make it match the network number.
Line 270: Line 179:
  
     export vxlan=64     export vxlan=64
-    export vxlan_name=ibnet0 # this doesn't get used by anything or stored anywhere 
     export local=172.20.64.9     export local=172.20.64.9
     export group=225.172.20.64     export group=225.172.20.64
Line 282: Line 190:
 //Et viola//, Ethernet on top of InfiniBand. //Et viola//, Ethernet on top of InfiniBand.
  
-If/when I get hardware capable of VXLAN offload, the dstport might have to change.+Note: The Infiniband MTU is 4k; IPoIB takes 4 bytes and VXLAN takes 50, leaving a max MTU of 4042.
  
 ====Multimedia==== ====Multimedia====
Line 288: Line 196:
  
 I also want to throw audio frames around with "no latency added". Someday, someday, someday. I also want to throw audio frames around with "no latency added". Someday, someday, someday.
 +
 +====GUIDs====
 +GUIDs and MAC addresses for Mellanox cards are listed on their respective pages. These two are ancient Qlogic adapters. I've never gotten them to work on my network; I pretty much just bought them to destroy and learn.
 +
 +  * 001175000079b560 - uninstalled qib
 +  * 001175000079b856 - uninstalled qib
 +
nndocs/infiniband.1734199140.txt.gz · Last modified: 2024/12/14 17:59 by naptastic