Naptastic

User Tools

Site Tools

Trace: • nvme-of • infiniband • srp • sandbox
nndocs:srp

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
nndocs:srp [2025/04/15 18:42] – naptasticnndocs:srp [2025/11/08 06:12] (current) – target/ACL confusion reduction naptastic
Line 14: Line 14:
   fe800000000000005849560e59150301   fe800000000000005849560e59150301
  
-The top 64 bits of the ACL might need to be 0 or they might be the same as the GUID; I don't know the initiator ID actually gets generated and I'm not doing any more digging into it.+Initiator ACLs start with all 0's. Targets start with fe80.
  
   /srpt> ib.fe800000000000005849560e53b70b01/acls create ib.00000000000000005849560e59150301   /srpt> ib.fe800000000000005849560e53b70b01/acls create ib.00000000000000005849560e59150301
Line 20: Line 20:
   Created mapped LUN 0.   Created mapped LUN 0.
  
-A Linux SRP target is always visible from all InfiniBand partitions. That limits its usefulness to me.+A Linux SRP target is always visible from all InfiniBand partitions.
  
 ====Dependencies==== ====Dependencies====
  
   apt install srptools   apt install srptools
 +
 +Do **NOT** set srp_daemon loose without using the -o flag! It will flood dmesg on both the initiator and the target!
  
 Find targets to connect to: Find targets to connect to:
Line 34: Line 36:
   * -c means "emit target information in a format we can use later"   * -c means "emit target information in a format we can use later"
   * -p 1 means "only scan on HCA port 1" so obviously change this if you are initiating from port 2...   * -p 1 means "only scan on HCA port 1" so obviously change this if you are initiating from port 2...
 +
 +====Configuration====
 +
 +It is **critical** that you edit /etc/srp_daemon.conf as soon as you have a list of targets and disallow connections to anything except the targets you want. The default file is well commented.
 +
 +To connect to a target listed by srp_daemon, write it to the appropriate add_target file in /sys/class/infiniband_srp. Here's how shark gets its swap ramdisk from southpark:
 +
 +  echo 'id_ext=5849560e53b70b01,ioc_guid=5849560e53b70b01,dgid=fe800000000000005849560e53b70b01,pkey=ffff,service_id=5849560e53b70b01' > /sys/class/infiniband_srp/srp-ibp14s0f0-1/add_target
  
 Lazy benchmarking seems good: Lazy benchmarking seems good:
Line 48: Line 58:
   17179869184 bytes (17 GB, 16 GiB) copied, 13.7431 s, 1.3 GB/s   17179869184 bytes (17 GB, 16 GiB) copied, 13.7431 s, 1.3 GB/s
  
-...so I'm using some for swap. We'll see how long it takes me to induce a kernel panic. ;)+====Logout==== 
 + 
 +"Delete the port" sounds pretty destructive, but this actually is the graceful way to close the connection. 
 + 
 +  # echo 1 > /sys/class/srp_remote_ports/[tab tab tab]/delete
nndocs/srp.1744742564.txt.gz · Last modified: 2025/04/15 18:42 by naptastic