Differences

This shows you the differences between two versions of the page.

--- nndocs:infiniband [2024/12/24 20:07] – [Partitions: OpenSM configuration] fix thinko naptastic
+++ nndocs:infiniband [2026/07/21 21:55] (current) – [SR-IOV] don't try SR-IOV with ConnectX-3 naptastic
@@ Line 122: / Line 122: @@
 =====SR-IOV=====
+Do not try to use SR-IOV with ConnectX-3 cards. It doesn't work. These instructions are for the Connect-IB and ConnectX-4 cards.
 ====Hardware Settings====
 The BIOS needs to have SR-IOV, ARI, and ACS support enabled.
@@ Line 205: / Line 207: @@
   * See [[iSCSI]] for information about enabling and using iSER.
   * See [[NVMe-oF]] for using NVMe over InfiniBand.
-  * I'm curious about the performance of SRP. I have gotten it to work before but it's not working now. Also not a huge priority.
+  * [[SRP]] is fast and, in most other ways, sucks.
 ====Networking====
 ===VXLAN===
-VXLAN is not the only way to get an Ethernet device on Infiniband, but as far as I can tell it's the only decent one. Neither ConnectX-3 nor Connect-IB has VXLAN offload support.
+VXLAN is not the only way to get an Ethernet device on Infiniband, but as far as I can tell it's the only decent one. None of my hardware has VXLAN offload support.
   * VXLAN id can be anything from 0-16777215 inclusive. I make it match the network number.
@@ Line 218: / Line 220: @@
     export vxlan=64
-    export vxlan_name=ibnet0 # this doesn't get used by anything or stored anywhere
     export local=172.20.64.9
     export group=225.172.20.64
@@ Line 230: / Line 231: @@
 //Et viola//, Ethernet on top of InfiniBand.
-If/when I get hardware capable of VXLAN offload, the dstport might have to change.
+Note: The Infiniband MTU is 4k; IPoIB takes 4 bytes and VXLAN takes 50, leaving a max MTU of 4042.
 ====Multimedia====
 Yeah, someday I want to throw video frames around. There's an RFC or ISO standard for that, IIRC. There's also lgproxy, which is RDMA-aware.
@@ Line 240: / Line 240: @@
   * 5849560e59150301 - shark Connect-IB
   * 5849560e53b70b01 - southpark Connect-IB
-  * - duckling has a Connect-IB and I don't know its GUID
+  * 5849560e53660101 - duckling Connect-IB
   * 7cfe900300a0a080 - uninstalled Connect-IB
   * (there are several more uninstalled Connect-IB cards)