Naptastic

User Tools

Site Tools

Trace:
nndocs:infiniband

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
nndocs:infiniband [2026/08/08 13:45] – update slightly; add links for hardware naptasticnndocs:infiniband [2026/10/07 22:16] (current) – [Hardware Settings] naptastic
Line 102: Line 102:
  
 If you unset the high bit on the partition number (0x3128 instead of 0xb128) Linux will set the high bit before joining the partition. If OpenSM's configuration has that partition's membership set for "partial" or "both", the Linux host will not be able to connect to everything on that subnet, or possibly //anything// on that subnet, regardless of which value you use. If you unset the high bit on the partition number (0x3128 instead of 0xb128) Linux will set the high bit before joining the partition. If OpenSM's configuration has that partition's membership set for "partial" or "both", the Linux host will not be able to connect to everything on that subnet, or possibly //anything// on that subnet, regardless of which value you use.
- 
-It's worth finding out if Netplan can manage IB child interfaces. 
  
 ====Connected vs. Datagram==== ====Connected vs. Datagram====
Line 128: Line 126:
  
 =====SR-IOV===== =====SR-IOV=====
- 
-ConnectX-3 cards (mlx4*) handle SR-IOV completely differently. These instructions are for the Connect-IB and ConnectX-4 cards (mlx5*). 
  
 ====Hardware Settings==== ====Hardware Settings====
Line 140: Line 136:
     SRIOV_EN                                    True(1)     SRIOV_EN                                    True(1)
  
-FPP_EN (Function Per Port ENable) controls whether the card appears as two PCI devices, or as a single device with two ports. Under mlx4, every VF on a dual-port HCA has both ports, and NUM_OF_VFs is how many dual-port devices to create. Under mlx5, each port gets its own pool of VFs and NUM_OF_VFs is per-port.+FPP_EN (Function Per Port ENable) controls whether the card appears as two PCI devices, or as a single device with two ports. Under mlx5, each port gets its own pool of VFs and NUM_OF_VFs is per-port.
  
 I haven't tried large numbers of VFs. The hardware upper limit is 63 for Connect-IB and 127 for ConnectX-3. Any number of system components could impose lower limits. I don't expect to need so many guests with IOV networking anyway... I haven't tried large numbers of VFs. The hardware upper limit is 63 for Connect-IB and 127 for ConnectX-3. Any number of system components could impose lower limits. I don't expect to need so many guests with IOV networking anyway...
  
-To make VFs exist, put a number <= NUM_OF_VFS into sriov_numvfs for that device. Before doing so, I recommend turning off VF probing. Otherwise the VFs will all make IPoIB interfaces, which probably isn't what you want. This setting is per PF. +(Note added later) The ConnectX-3 cards can't actually do SR-IOV. On newer kernels (how new? Dunno) the mlx4_core driver's SR-IOV functionality does not work anymore. You can create virtual functions, assign them to VMs, whatever... no traffic will pass. There's a thread on ServeTheHome forums about it. I'm not hopeful that it will ever get fixed.
- +
-I'm still checking if there's a way to configure the driver so this becomes the default setting. +
- +
-  # echo 0 > /sys/class/infiniband/ibp13s0f0/device/sriov_drivers_autoprobe +
-  # echo 0 > /sys/class/infiniband/ibp13s0f1/device/sriov_drivers_autoprobe +
- +
-If it works, there will be new PCI devices as well as VFs listed under `ip link`: +
- +
-  # echo 7 > /sys/class/infiniband/ibp6s0f0/device/sriov_numvfs # no output on success; check dmesg for interesting but probably useless messages. +
-  # lspci | grep nfi +
-  06:00.0 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB] +
-  06:00.1 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB] +
-  06:00.2 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
-  06:00.3 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
-  06:00.4 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
-  06:00.5 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
-  06:00.6 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
-  06:00.7 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
-  06:01.0 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function] +
- +
-Warning: The output from ip link is very wide. +
- +
-  # ip link +
-  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 +
-      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 +
-  2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 +
-      link/ether 3c:ec:ef:6d:10:62 brd ff:ff:ff:ff:ff:ff +
-  3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 +
-      link/ether 3c:ec:ef:6d:10:63 brd ff:ff:ff:ff:ff:ff +
-  4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 qdisc fq_codel state UP mode DEFAULT group default qlen 256 +
-      link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff +
-      vf 0     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-      vf 1     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-      vf 2     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-      vf 3     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-      vf 4     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-      vf 5     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-      vf 6     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off +
-  5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc fq_codel state DOWN mode DEFAULT group default qlen 256 +
-      link/infiniband 80:00:00:28:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:09 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff +
- +
-====VF Configuration==== +
-The official documentation covers a sysfs interface for configuring VF properties. That interface hasn't existed for years. Before using a VF, you must set node_guid, port_guid, and state using ip link. Make port_guid == node_guid == unique. (I use the base port guid + VF + 1.) +
- +
-GUIDs need to be set before attaching a VF to a VM. It should be possible to change state (simulating unplugging the cable) while a VM is using a VF but I haven't tested this. +
- +
-I keep the whole configuration in /etc/rc.local. +
 =====Upper-Layer Protocols (ULPs)===== =====Upper-Layer Protocols (ULPs)=====
 RDMA opens all kinds of possibilities for RDMA-aware protocols to be amazing and fast. They probably all deserve their own pages. RDMA opens all kinds of possibilities for RDMA-aware protocols to be amazing and fast. They probably all deserve their own pages.
Line 250: Line 198:
 I also want to throw audio frames around with "no latency added". Someday, someday, someday. I also want to throw audio frames around with "no latency added". Someday, someday, someday.
  
-=====GUIDs===== +====GUIDs==== 
-====Installed==== +GUIDs and MAC addresses for Mellanox cards are listed on their respective pages. These two are ancient Qlogic adapters. I've never gotten them to work on my network; I pretty much just bought them to destroy and learn. 
-  * b8cef603004ebc86 - southpark ConnectX-4 FCAT +
-  * 0002c90300b37f10 - shark ConnectX-3 FCBT with no date on the label +
-  * 5849560e53660101 - duckling Connect-IB +
-====Not Installed==== +
-  * 5849560e53b70b01 - Connect-IB +
-  * 5849560e59150301 - Connect-IB +
-  * 7cfe900300a0a080 - Connect-IB +
-  * (there are several more uninstalled Connect-IB cards) +
-  * f4521403002c18b0 - uninstalled ConnectX-3 dated 2014-01-29+
   * 001175000079b560 - uninstalled qib   * 001175000079b560 - uninstalled qib
   * 001175000079b856 - uninstalled qib   * 001175000079b856 - uninstalled qib
  
nndocs/infiniband.1786196745.txt.gz · Last modified: 2026/08/08 13:45 by naptastic