Naptastic

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====targetcli-fb====
  * it needs to be turned on in the kernel. drivers -> Infiniband -> SRP
  * turn on both target and initiator please
  * in targetcli, /srpt is the relevant path
  * targets are formatted like ib.fe80000000000000f4521403002c18b1
  * idk how you get those numbers except to examine an ipoib link, such as:

  5: ib0: <...> mtu 2044 ...
      link/infiniband 80:00:02:18:fe:80:00:00:00:00:00:00:f4:52:14:03:00:2c:18:b1
      brd             00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

Now strip the first 4 bytes off (they change anyway) and remove the :'s

  fe800000000000005849560e59150301

Initiator ACLs start with all 0's. Targets start with fe80.

  /srpt> ib.fe800000000000005849560e53b70b01/acls create ib.00000000000000005849560e59150301
  Created Node ACL for ib.00000000000000005849560e59150301
  Created mapped LUN 0.

A Linux SRP target is always visible from all InfiniBand partitions.

====Dependencies====

  apt install srptools

Do **NOT** set srp_daemon loose without using the -o flag! It will flood dmesg on both the initiator and the target!

Find targets to connect to:

    # srp_daemon -o -v -c -p 1

  * -o means "run once" otherwise dmesg on all your hosts will get polluted with SRP login noise.
  * -v means "say what you're doing"
  * -c means "emit target information in a format we can use later"
  * -p 1 means "only scan on HCA port 1" so obviously change this if you are initiating from port 2...

Note: in my experimentation, port 2 is not actually usable. A target created on a dual-port card always answers on port 1 even if the associated GUID is for port 2, and an initiator on port 2 always actually tries to use port 1. The answer is probably "use SR-IOV".
====Configuration====

It is **critical** that you edit /etc/srp_daemon.conf as soon as you have a list of targets and disallow connections to anything except the targets you want. The default file is well commented.

To connect to a target listed by srp_daemon, write it to the appropriate add_target file in /sys/class/infiniband_srp. Here's how shark gets its swap ramdisk from southpark:

  [root]@[shark][~]# echo 'id_ext=5849560e53b70b01,ioc_guid=5849560e53b70b01,dgid=fe800000000000005849560e53b70b01,pkey=ffff,service_id=5849560e53b70b01' > \
  /sys/class/infiniband_srp/srp-ibp14s0f0-1/add_target
  [root]@[shark][~]# dmesg
    (...snip...)
  [2719206.378801] scsi host8: SRP.T10:5849560E53B70B01
  [2719206.379439] scsi 8:0:0:0: Direct-Access     LIO-ORG  swap             4.0  PQ: 0 ANSI: 6
  [2719206.380206] sd 8:0:0:0: Attached scsi generic sg5 type 0
  [2719206.380337] sd 8:0:0:0: [sdd] 33554432 512-byte logical blocks: (17.2 GB/16.0 GiB)
  [2719206.380376] scsi host8: ib_srp: new target: id_ext 5849560e53b70b01 ioc_guid 5849560e53b70b01 pkey ffff service_id 5849560e53b70b01 sgid fe80:0000:0000:0000:5849:560e:5915:0301 dgid fe80:0000:0000:0000:5849:560e:53b7:0b01
  [2719206.380380] sd 8:0:0:0: [sdd] Write Protect is off
  [2719206.380384] sd 8:0:0:0: [sdd] Mode Sense: 43 00 00 08
  [2719206.380452] sd 8:0:0:0: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
  [2719206.395027] sd 8:0:0:0: [sdd] Preferred minimum I/O size 512 bytes
  [2719206.395030] sd 8:0:0:0: [sdd] Optimal transfer size 4294967288 logical blocks > dev_max (65535 logical blocks)
  [2719206.425809] sd 8:0:0:0: [sdd] Attached SCSI disk

Lazy benchmarking seems good:

  [root]@[shark][~]# dd if=/dev/sdb of=/dev/null bs=4M
  4096+0 records in
  4096+0 records out
  17179869184 bytes (17 GB, 16 GiB) copied, 5.38771 s, 3.2 GB/s
  
  [root]@[shark][~]# dd if=/dev/zero of=/dev/sdb bs=4M
  dd: error writing '/dev/sdb': No space left on device
  4097+0 records in
  4096+0 records out
  17179869184 bytes (17 GB, 16 GiB) copied, 13.7431 s, 1.3 GB/s

====Logout====

"Delete the port" sounds pretty destructive, but this actually is the graceful way to close the connection.

  # echo 1 > /sys/class/srp_remote_ports/ [tab tab tab] /delete