Differences

This shows you the differences between two versions of the page.

--- nndocs:srp [2025/02/24 18:25] – naptastic
+++ nndocs:srp [2025/11/08 06:12] (current) – target/ACL confusion reduction naptastic
@@ Line 5: / Line 5: @@
   * targets are formatted like ib.fe80000000000000f4521403002c18b1
   * idk how you get those numbers except to examine an ipoib link, such as:
-  (reformatted slightly)
 : ib0: <...> mtu 2044 ...
@@ Line 13: / Line 11: @@
 Now strip the first 4 bytes off (they change anyway) and remove the :'s
+  fe800000000000005849560e59150301
+Initiator ACLs start with all 0's. Targets start with fe80.
+  /srpt> ib.fe800000000000005849560e53b70b01/acls create ib.00000000000000005849560e59150301
+  Created Node ACL for ib.00000000000000005849560e59150301
+  Created mapped LUN 0.
+A Linux SRP target is always visible from all InfiniBand partitions.
 ====Dependencies====
@@ Line 18: / Line 26: @@
   apt install srptools
-===Login fails===
+Do **NOT** set srp_daemon loose without using the -o flag! It will flood dmesg on both the initiator and the target!
+Find targets to connect to:
+    # srp_daemon -o -v -c -p 1
+  * -o means "run once" otherwise dmesg on all your hosts will get polluted with SRP login noise.
+  * -v means "say what you're doing"
+  * -c means "emit target information in a format we can use later"
+  * -p 1 means "only scan on HCA port 1" so obviously change this if you are initiating from port 2...
+====Configuration====
+It is **critical** that you edit /etc/srp_daemon.conf as soon as you have a list of targets and disallow connections to anything except the targets you want. The default file is well commented.
+To connect to a target listed by srp_daemon, write it to the appropriate add_target file in /sys/class/infiniband_srp. Here's how shark gets its swap ramdisk from southpark:
+  echo 'id_ext=5849560e53b70b01,ioc_guid=5849560e53b70b01,dgid=fe800000000000005849560e53b70b01,pkey=ffff,service_id=5849560e53b70b01' > /sys/class/infiniband_srp/srp-ibp14s0f0-1/add_target
-shark (initiator):
+Lazy benchmarking seems good:
-   [74794.509035] scsi host11: ib_srp: REJ received
+  [root]@[shark][~]# dd if=/dev/sdb of=/dev/null bs=4M
-   [74794.509038] scsi host11: ib_srp: SRP LOGIN from fe80:0000:0000:0000:5849:560e:5915:0301 to fe80:0000:0000:0000:5849:560e:53b7:0b09 REJECTED, reason 0x00010001
++0 records in
-   [74794.509048] scsi host11: ib_srp: Connection 0/12 to fe80:0000:0000:0000:5849:560e:53b7:0b09 failed
++0 records out
+  17179869184 bytes (17 GB, 16 GiB) copied, 5.38771 s, 3.2 GB/s
+  [root]@[shark][~]# dd if=/dev/zero of=/dev/sdb bs=4M
+  dd: error writing '/dev/sdb': No space left on device
++0 records in
++0 records out
+  17179869184 bytes (17 GB, 16 GiB) copied, 13.7431 s, 1.3 GB/s
-southpark (target):
+====Logout====
-   [4483481.918835] ib_srpt Received SRP_LOGIN_REQ with i_port_id 5849:560e:53b7:0b09:5849:560e:5915:0301, t_port_id 5849:560e:53b7:0b01:5849:560e:53b7:0b01 and it_iu_len 8260 on port 1 (guid=fe80:0000:0000:0000:5849:560e:53b7:0b09); pkey 0xb068
+"Delete the port" sounds pretty destructive, but this actually is the graceful way to close the connection.
-   [4483481.945664] ib_srpt rejected SRP_LOGIN_REQ because target port ibp33s0f1_1 has not yet been enabled
-   [4483481.959142] ib_srpt Rejecting login with reason 0x10001
+  # echo 1 > /sys/class/srp_remote_ports/[tab tab tab]/delete