Differences

This shows you the differences between two versions of the page.

--- nndocs:srp [2025/11/08 05:52] – add logout info naptastic
+++ nndocs:srp [2026/06/11 17:57] (current) – update commands and "port 2 doesn't work" info naptastic
@@ Line 14: / Line 14: @@
   fe800000000000005849560e59150301
-The top 64 bits of the ACL might need to be 0 or they might be the same as the GUID; I don't know the initiator ID actually gets generated and I'm not doing any more digging into it.
+Initiator ACLs start with all 0's. Targets start with fe80.
   /srpt> ib.fe800000000000005849560e53b70b01/acls create ib.00000000000000005849560e59150301
@@ Line 20: / Line 20: @@
   Created mapped LUN 0.
-A Linux SRP target is always visible from all InfiniBand partitions. That limits its usefulness to me.
+A Linux SRP target is always visible from all InfiniBand partitions.
 ====Dependencies====
@@ Line 30: / Line 30: @@
 Find targets to connect to:
-    # srp_daemon -o -v -c -p 1
+    # srp_daemon -o -V -c
   * -o means "run once" otherwise dmesg on all your hosts will get polluted with SRP login noise.
-  * -v means "say what you're doing"
+  * -V means debug. This will give you the id_ext (lower 8 bytes) of every target.
   * -c means "emit target information in a format we can use later"
-  * -p 1 means "only scan on HCA port 1" so obviously change this if you are initiating from port 2...
+  * If all the information returned by srp_daemon is correct, you can replace -c with -e to connect to targets automatically. Otherwise, you can connect to individual targets manually using the steps below.
+Note: in my experimentation, port 2 is not actually usable. A target created a dual-port InfiniBand HCA always answers on port 1 even if the associated GUID is for port 2, and an initiator on port 2 always actually tries to use port 1. The answer is probably "use SR-IOV".
+On VPI cards configured IB/ETH, port 2 doesn't support SRP anyway.
 ====Configuration====
@@ Line 43: / Line 47: @@
 To connect to a target listed by srp_daemon, write it to the appropriate add_target file in /sys/class/infiniband_srp. Here's how shark gets its swap ramdisk from southpark:
-  echo 'id_ext=5849560e53b70b01,ioc_guid=5849560e53b70b01,dgid=fe800000000000005849560e53b70b01,pkey=ffff,service_id=5849560e53b70b01' > /sys/class/infiniband_srp/srp-ibp14s0f0-1/add_target
+  [root]@[shark][~]# echo 'id_ext=5849560e53b70b01,ioc_guid=5849560e53b70b01,dgid=fe800000000000005849560e53b70b01,pkey=ffff,service_id=5849560e53b70b01' > \
+  /sys/class/infiniband_srp/srp-ibp14s0f0-1/add_target
+  [root]@[shark][~]# dmesg
+    (...snip...)
+  [2719206.378801] scsi host8: SRP.T10:5849560E53B70B01
+  [2719206.379439] scsi 8:0:0:0: Direct-Access     LIO-ORG  swap             4.0  PQ: 0 ANSI: 6
+  [2719206.380206] sd 8:0:0:0: Attached scsi generic sg5 type 0
+  [2719206.380337] sd 8:0:0:0: [sdd] 33554432 512-byte logical blocks: (17.2 GB/16.0 GiB)
+  [2719206.380376] scsi host8: ib_srp: new target: id_ext 5849560e53b70b01 ioc_guid 5849560e53b70b01 pkey ffff service_id 5849560e53b70b01 sgid fe80:0000:0000:0000:5849:560e:5915:0301 dgid fe80:0000:0000:0000:5849:560e:53b7:0b01
+  [2719206.380380] sd 8:0:0:0: [sdd] Write Protect is off
+  [2719206.380384] sd 8:0:0:0: [sdd] Mode Sense: 43 00 00 08
+  [2719206.380452] sd 8:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
+  [2719206.395027] sd 8:0:0:0: [sdd] Preferred minimum I/O size 512 bytes
+  [2719206.395030] sd 8:0:0:0: [sdd] Optimal transfer size 4294967288 logical blocks > dev_max (65535 logical blocks)
+  [2719206.425809] sd 8:0:0:0: [sdd] Attached SCSI disk
 Lazy benchmarking seems good:
@@ Line 62: / Line 80: @@
 "Delete the port" sounds pretty destructive, but this actually is the graceful way to close the connection.
-  # echo 1 > /sys/class/srp_remote_ports/[tab tab tab]/delete
+  # echo 1 > /sys/class/srp_remote_ports/ [tab tab tab] /delete