Naptastic

• it needs to be turned on in the kernel. drivers → Infiniband → SRP
• turn on both target and initiator please
• in targetcli, /srpt is the relevant path
• targets are formatted like ib.fe80000000000000f4521403002c18b1
• idk how you get those numbers except to examine an ipoib link, such as:

5: ib0: <...> mtu 2044 ...
    link/infiniband 80:00:02:18:fe:80:00:00:00:00:00:00:f4:52:14:03:00:2c:18:b1
    brd             00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

Now strip the first 4 bytes off (they change anyway) and remove the :'s

fe800000000000005849560e59150301

The top 64 bits of the ACL might need to be 0 or they might be the same as the GUID; I don't know the initiator ID actually gets generated and I'm not doing any more digging into it.

/srpt> ib.fe800000000000005849560e53b70b01/acls create ib.00000000000000005849560e59150301
Created Node ACL for ib.00000000000000005849560e59150301
Created mapped LUN 0.

A Linux SRP target is always visible from all InfiniBand partitions. That limits its usefulness to me.

apt install srptools

Do NOT set srp_daemon loose without using the -o flag! It will flood dmesg on both the initiator and the target!

Find targets to connect to:

  # srp_daemon -o -v -c -p 1

• -o means “run once” otherwise dmesg on all your hosts will get polluted with SRP login noise.
• -v means “say what you're doing”
• -c means “emit target information in a format we can use later”
• -p 1 means “only scan on HCA port 1” so obviously change this if you are initiating from port 2…

It is critical that you edit /etc/srp_daemon.conf as soon as you have a list of targets and disallow connections to anything except the targets you want. The default file is well commented.

To connect to a target listed by srp_daemon, write it to the appropriate add_target file in /sys/class/infiniband_srp. Here's how shark gets its swap ramdisk from southpark:

echo 'id_ext=5849560e53b70b01,ioc_guid=5849560e53b70b01,dgid=fe800000000000005849560e53b70b01,pkey=ffff,service_id=5849560e53b70b01' > /sys/class/infiniband_srp/srp-ibp14s0f0-1/add_target

Lazy benchmarking seems good:

[root]@[shark][~]# dd if=/dev/sdb of=/dev/null bs=4M
4096+0 records in
4096+0 records out
17179869184 bytes (17 GB, 16 GiB) copied, 5.38771 s, 3.2 GB/s

[root]@[shark][~]# dd if=/dev/zero of=/dev/sdb bs=4M
dd: error writing '/dev/sdb': No space left on device
4097+0 records in
4096+0 records out
17179869184 bytes (17 GB, 16 GiB) copied, 13.7431 s, 1.3 GB/s

“Delete the port” sounds pretty destructive, but this actually is the graceful way to close the connection.

# echo 1 > /sys/class/srp_remote_ports/[tab tab tab]/delete