Differences

This shows you the differences between two versions of the page.

--- nndocs:srp [2025/02/14 12:56] – created naptastic
+++ nndocs:srp [2025/04/15 18:42] (current) – naptastic
@@ Line 1: / Line 1: @@
 ====targetcli-fb====
-The documentation for SRP is woefully inadequte.
+  * it needs to be turned on in the kernel. drivers -> Infiniband -> SRP
+  * turn on both target and initiator please
+  * in targetcli, /srpt is the relevant path
+  * targets are formatted like ib.fe80000000000000f4521403002c18b1
+  * idk how you get those numbers except to examine an ipoib link, such as:
-  /srpt> ib.fe800000000000005849560e53b70b01/acls create
+: ib0: <...> mtu 2044 ...
-  Missing required parameter wwn
+      link/infiniband 80:00:02:18:fe:80:00:00:00:00:00:00:f4:52:14:03:00:2c:18:b1
-  /srpt> ib.fe800000000000005849560e53b70b01/acls create wwn=ib.800000000000005849560e53b70b01
+      brd             00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
-  WWN not valid as: ib
-  /srpt> ib.fe800000000000005849560e53b70b01/acls create wwn=800000000000005849560e53b70b01
-  WWN not valid as: ib
-From man targetcli(8):
+Now strip the first 4 bytes off (they change anyway) and remove the :'s
-  CONFIGURING SRP
+  fe800000000000005849560e59150301
-         SRP (SCSI RDMA Protocol) requires that RDMA-capable hardware is present. It uses "ib" WWNs.
-NONE OF THIS TELLS ME HOW TO FORMAT THE STRING. Let's go spelunking in the kernel, I guess...
+The top 64 bits of the ACL might need to be 0 or they might be the same as the GUID; I don't know the initiator ID actually gets generated and I'm not doing any more digging into it.
+  /srpt> ib.fe800000000000005849560e53b70b01/acls create ib.00000000000000005849560e59150301
+  Created Node ACL for ib.00000000000000005849560e59150301
+  Created mapped LUN 0.
+A Linux SRP target is always visible from all InfiniBand partitions. That limits its usefulness to me.
+====Dependencies====
+  apt install srptools
+Find targets to connect to:
+    # srp_daemon -o -v -c -p 1
+  * -o means "run once" otherwise dmesg on all your hosts will get polluted with SRP login noise.
+  * -v means "say what you're doing"
+  * -c means "emit target information in a format we can use later"
+  * -p 1 means "only scan on HCA port 1" so obviously change this if you are initiating from port 2...
+Lazy benchmarking seems good:
+  [root]@[shark][~]# dd if=/dev/sdb of=/dev/null bs=4M
++0 records in
++0 records out
+  17179869184 bytes (17 GB, 16 GiB) copied, 5.38771 s, 3.2 GB/s
+  [root]@[shark][~]# dd if=/dev/zero of=/dev/sdb bs=4M
+  dd: error writing '/dev/sdb': No space left on device
++0 records in
++0 records out
+  17179869184 bytes (17 GB, 16 GiB) copied, 13.7431 s, 1.3 GB/s
+...so I'm using some for swap. We'll see how long it takes me to induce a kernel panic. ;)