Naptastic

User Tools

Site Tools

Trace:
nndocs:srp

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
nndocs:srp [2025/04/15 18:42] – naptasticnndocs:srp [2026/06/11 17:57] (current) – update commands and "port 2 doesn't work" info naptastic
Line 14: Line 14:
   fe800000000000005849560e59150301   fe800000000000005849560e59150301
  
-The top 64 bits of the ACL might need to be 0 or they might be the same as the GUID; I don't know the initiator ID actually gets generated and I'm not doing any more digging into it.+Initiator ACLs start with all 0's. Targets start with fe80.
  
   /srpt> ib.fe800000000000005849560e53b70b01/acls create ib.00000000000000005849560e59150301   /srpt> ib.fe800000000000005849560e53b70b01/acls create ib.00000000000000005849560e59150301
Line 20: Line 20:
   Created mapped LUN 0.   Created mapped LUN 0.
  
-A Linux SRP target is always visible from all InfiniBand partitions. That limits its usefulness to me.+A Linux SRP target is always visible from all InfiniBand partitions.
  
 ====Dependencies==== ====Dependencies====
  
   apt install srptools   apt install srptools
 +
 +Do **NOT** set srp_daemon loose without using the -o flag! It will flood dmesg on both the initiator and the target!
  
 Find targets to connect to: Find targets to connect to:
  
-    # srp_daemon -o -v -c -p 1+    # srp_daemon -o -V -c
  
   * -o means "run once" otherwise dmesg on all your hosts will get polluted with SRP login noise.   * -o means "run once" otherwise dmesg on all your hosts will get polluted with SRP login noise.
-  * -v means "say what you're doing"+  * -V means debug. This will give you the id_ext (lower 8 bytes) of every target.
   * -c means "emit target information in a format we can use later"   * -c means "emit target information in a format we can use later"
-  * -p 1 means "only scan on HCA port 1" so obviously change this if you are initiating from port 2...+  * If all the information returned by srp_daemon is correct, you can replace -c with -e to connect to targets automatically. Otherwise, you can connect to individual targets manually using the steps below. 
 + 
 +Note: in my experimentation, port 2 is not actually usable. A target created a dual-port InfiniBand HCA always answers on port 1 even if the associated GUID is for port 2, and an initiator on port 2 always actually tries to use port 1. The answer is probably "use SR-IOV". 
 + 
 +On VPI cards configured IB/ETH, port 2 doesn't support SRP anyway. 
 + 
 +====Configuration==== 
 + 
 +It is **critical** that you edit /etc/srp_daemon.conf as soon as you have a list of targets and disallow connections to anything except the targets you want. The default file is well commented. 
 + 
 +To connect to a target listed by srp_daemon, write it to the appropriate add_target file in /sys/class/infiniband_srp. Here's how shark gets its swap ramdisk from southpark: 
 + 
 +  [root]@[shark][~]# echo 'id_ext=5849560e53b70b01,ioc_guid=5849560e53b70b01,dgid=fe800000000000005849560e53b70b01,pkey=ffff,service_id=5849560e53b70b01' > \ 
 +  /sys/class/infiniband_srp/srp-ibp14s0f0-1/add_target 
 +  [root]@[shark][~]# dmesg 
 +    (...snip...) 
 +  [2719206.378801] scsi host8: SRP.T10:5849560E53B70B01 
 +  [2719206.379439] scsi 8:0:0:0: Direct-Access     LIO-ORG  swap             4.0  PQ: 0 ANSI: 6 
 +  [2719206.380206] sd 8:0:0:0: Attached scsi generic sg5 type 0 
 +  [2719206.380337] sd 8:0:0:0: [sdd] 33554432 512-byte logical blocks: (17.2 GB/16.0 GiB) 
 +  [2719206.380376] scsi host8: ib_srp: new target: id_ext 5849560e53b70b01 ioc_guid 5849560e53b70b01 pkey ffff service_id 5849560e53b70b01 sgid fe80:0000:0000:0000:5849:560e:5915:0301 dgid fe80:0000:0000:0000:5849:560e:53b7:0b01 
 +  [2719206.380380] sd 8:0:0:0: [sdd] Write Protect is off 
 +  [2719206.380384] sd 8:0:0:0: [sdd] Mode Sense: 43 00 00 08 
 +  [2719206.380452] sd 8:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA 
 +  [2719206.395027] sd 8:0:0:0: [sdd] Preferred minimum I/O size 512 bytes 
 +  [2719206.395030] sd 8:0:0:0: [sdd] Optimal transfer size 4294967288 logical blocks > dev_max (65535 logical blocks) 
 +  [2719206.425809] sd 8:0:0:0: [sdd] Attached SCSI disk
  
 Lazy benchmarking seems good: Lazy benchmarking seems good:
Line 48: Line 76:
   17179869184 bytes (17 GB, 16 GiB) copied, 13.7431 s, 1.3 GB/s   17179869184 bytes (17 GB, 16 GiB) copied, 13.7431 s, 1.3 GB/s
  
-...so I'm using some for swap. We'll see how long it takes me to induce a kernel panic. ;)+====Logout==== 
 + 
 +"Delete the port" sounds pretty destructive, but this actually is the graceful way to close the connection. 
 + 
 +  # echo 1 > /sys/class/srp_remote_ports/ [tab tab tab] /delete
nndocs/srp.1744742564.txt.gz · Last modified: 2025/04/15 18:42 by naptastic