Differences

This shows you the differences between two versions of the page.

--- nndocs:infiniband [2024/12/19 18:52] – [VF Configuration] keep VF config in /etc/rc.local naptastic
+++ nndocs:infiniband [2025/01/21 14:38] (current) – [Networking] correct a thing naptastic
@@ Line 78: / Line 78: @@
 Here's a block for my ATA over Ethernet experiments. Subject to change. IP addresses are necessary for setting up VXLAN tunnels. Checking if IPv6 tunnels perform differently from IPv4 tunnels is on the to-do list. I suspect they perform better. Needs testing.
-  mp1=0x3128, ipoib, rate=12, mtu=5, scope=2, defmember=full:
+  storage=0xb128, ipoib, rate=12, mtu=5, scope=2, defmember=full:
       mgid=ff12:401b::ffff:ffff   # IPv4 Broadcast address
       mgid=ff12:401b::1           # IPv4 All Hosts group
@@ Line 85: / Line 85: @@
 ====Partitions: Host configuration====
 There's no functional netlink interface for creating child interfaces. You must use the sysfs interface.
-  # echo 0xb129 > /sys/class/net/ib0/create_child
+  # echo 0xb128 > /sys/class/net/ib0/create_child
-The sysfs interface for deleting child interfaces doesn't work (for me at least). You must use the netlink interface.
-  # ip link del ib0.b129
 Resist the temptation to rename the interface to something descriptive. **It's already self-descriptive**. Creative naming is for VXLAN tunnels and bridges, e.g.:
-  # ip link add vx129 type vxlan id 129 local 172.20.129.9 group 225.172.20.129
+  # ip link add vx128 type vxlan id 128 local 172.20.128.13 group 225.172.20.128
-  # ip link set master aoe2 dev vx129
+  # ip link set master aoe1 dev vx128
+The sysfs interface for deleting child interfaces doesn't work (for me at least). You must use the netlink interface.
+  # ip link del ib0.b128
-If you unset the high bit on the partition number (0x3129 instead of 0xb129) Linux will set the high bit before joining the partition. If OpenSM's configuration has that partition's membership set for "partial" or "both", the Linux host will not be able to connect to everything on that subnet, or possibly //anything// on that subnet, regardless of which value you use.
+If you unset the high bit on the partition number (0x3128 instead of 0xb128) Linux will set the high bit before joining the partition. If OpenSM's configuration has that partition's membership set for "partial" or "both", the Linux host will not be able to connect to everything on that subnet, or possibly //anything// on that subnet, regardless of which value you use.
 It's worth finding out if Netplan can manage IB child interfaces.
@@ Line 208: / Line 208: @@
 ====Networking====
 ===VXLAN===
-VXLAN is not the only way to get an Ethernet device on Infiniband, but as far as I can tell it's the only decent one. Neither ConnectX-3 nor Connect-IB has VXLAN offload support. Despite this, in connected mode, VXLAN is still really fast.
+VXLAN is not the only way to get an Ethernet device on Infiniband, but as far as I can tell it's the only decent one. None of my hardware has VXLAN offload support.
   * VXLAN id can be anything from 0-16777215 inclusive. I make it match the network number.
@@ Line 236: / Line 236: @@
 I also want to throw audio frames around with "no latency added". Someday, someday, someday.
+=====GUIDs=====
+  * 5849560e59150301 - shark Connect-IB
+  * 5849560e53b70b01 - southpark Connect-IB
+  * 5849560e53660101 - duckling Connect-IB
+  * 7cfe900300a0a080 - uninstalled Connect-IB
+  * (there are several more uninstalled Connect-IB cards)
+  * f4521403002c18b0 - uninstalled ConnectX-3 2014-01-29
+  * 0002c90300b37f10 - uninstalled ConnectX-3 with no date on the label
+  * 001175000079b560 - uninstalled qib
+  * 001175000079b856 - uninstalled qib