Naptastic

This is an old revision of the document!

TODO Please put storage traffic on its own IB subnet and limit things sanely.

NQNs: nqn.2014-08.rocks.narf.hostname is the whole thing. No :01 or any of that jazz, at least not yet.

The goal is to run everything over RDMA (InfiniBand specifically) and maybe have TCP as a fallback. (If TCP, then I probably want to try VMA… which I don't want to do.)

Target

NVMe calls targets “subsystems”. Good for them.

Create the subsystem, port and properties:

  /> subsystems/ create nqn.2014-08.rocks.narf.southpark
  /> hosts/ create nqn.2014-08.rocks.narf.sadness
  /> ports/ create 1
  /> ports/1/ set addr trtype=rdma
  Parameter trtype is now 'rdma'.
  /> ports/1/ set addr adrfam=ipv4
  Parameter adrfam is now 'ipv4'.
  /> ports/1/ set addr traddr=172.20.64.13
  Parameter traddr is now '172.20.64.13'.
  /> ports/1/ set addr trsvcid=4420
  Parameter trsvcid is now '4420'.
  /> ports/1/subsystems create nqn.2014-08.rocks.narf.southpark

NVMe uses “namespaces” (which are numbers) instead of LUNs. Good for them.

Create and share namespace:

  /> subsystems/nqn.2014-08.rocks.narf.southpark/namespaces create 1
  /> subsystems/nqn.2014-08.rocks.narf.southpark/namespaces/1 set device path=/dev/nvme0n1
  Parameter path is now '/dev/nvme0n1'.
  /> subsystems/nqn.2014-08.rocks.narf.southpark/namespaces/1 enable
  The Namespace has been enabled.

Create ACLs:

  /> subsystems/nqn.2014-08.rocks.narf.southpark/ set attr allow_any_host=0
  Parameter allow_any_host is now '0'.
  /> subsystems/nqn.2014-08.rocks.narf.southpark/allowed_hosts create nqn.2014-08.rocks.narf.sadness

Initiator

NVMe calls initiators “hosts”. Good for them.

  [root]@[shark][13:07:44][~]# modprobe nvme-rdma
  [root]@[shark][08:42:11][~]# nvme discover -t rdma -a 172.20.64.13 -s 4420
  Failed to write to /dev/nvme-fabrics: Connection reset by peer
  failed to add controller, error failed to write to nvme-fabrics device

And on southpark:

  [956501.535293] nvme nvme1: Connect rejected: status 8 (invalid service ID).
  [956501.535302] nvme nvme1: rdma connection establishment failed (-104)

Ok, let's try this from sadness. Target configuration (nvmetcli) :

  /hosts> cd /
  /> hosts/ create nqn.2014-08.rocks.narf.sadness
  /> subsystems/nqn.2014-08.rocks.narf.southpark/allowed_hosts/ create nqn.2014-08.rocks.narf.sadness

This doesn't work from inside nvmetcli. The newline character grunks up the whole thing.

  # echo -n ipv4 > /sys/kernel/config/nvmet/ports/1/addr_adrfam

On sadness:

  root@sadness:~# nvme discover -t rdma -a 172.20.64.13 -s 4420
  Failed to write to /dev/nvme-fabrics: Connection reset by peer

from dmesg on sadness:

  [88648.940995] nvme nvme1: rdma connection establishment failed (-104)