Naptastic

This is an old revision of the document!

TODO Please put storage traffic on its own IB subnet and limit things sanely.

The goal is to run everything over RDMA (InfiniBand specifically) and maybe have TCP as a fallback. (If TCP, then I probably want to try VMA… which I don't want to do.)

Target

NVMe calls targets “subsystems”. Good for them.

Create the subsystem:

  /> subsystems/ create nqn.2014-08.rocks.narf.southpark

NVMe calls TCP ports “service IDs”. Good for them.

Create a port and set its properties:

  /> ports/ create 1
  /> ports/1/ set addr trtype=rdma
  /> ports/1/ set addr adrfam=ipv4
  /> ports/1/ set addr traddr=172.20.64.13
  /> ports/1/ set addr trsvcid=4420
  /> ports/1/subsystems create nqn.2014-08.rocks.narf.southpark

NVMe calls LUNs “namespaces” (which are numbers). Good for them.

Create, set, and enable namespace:

  /> subsystems/nqn.2014-08.rocks.narf.southpark/namespaces create 1
  /> subsystems/nqn.2014-08.rocks.narf.southpark/namespaces/1 set device path=/dev/disk/by-id/nvme-eui.foo
  /> subsystems/nqn.2014-08.rocks.narf.southpark/namespaces/1 enable

Don't use /dev/nvme* names; they can change order on reboot.

You can use a file as a backstore. The syntax is not at all obvious. Why “group=device”? Whatever:

  />subsystems/nqn.2014-08.rocks.narf.hostname/namespaces/1 set group=device path=/path/to/some/host.img

I haven't checked if this works for partitions or SATA drives, but I think it should?

I haven't figured out ramdisks yet. There isn't a built-in system like in LIO.

Create ACLs:

  /> hosts/ create nqn.2014-08.rocks.narf.sadness
  /> subsystems/nqn.2014-08.rocks.narf.southpark/ set attr allow_any_host=0
  Parameter allow_any_host is now '0'.
  /> subsystems/nqn.2014-08.rocks.narf.southpark/allowed_hosts create nqn.2014-08.rocks.narf.sadness

Initiator

NVMe calls initiators “hosts”. Good for them.

Load the module (should be able to list this in /etc/modules)

  # modprobe nvme-rdma

Discovery:

  # nvme discover -t rdma -a 172.20.64.13 -s 4420

Log in:

  # nvme connect -t rdma -n nqn.2014-08.rocks.narf.southpark -a 172.20.64.13 -s 4420

Rescan:

  # nvme ns-rescan /dev/nvme1

If a namespace you expect to appear isn't appearing, check the target to see if you forget to enable it

Disconnect a subsystem:

  # nvme disconnect -d /dev/nvme1

Gotta admit, that feels a lot nicer than logging out of iscsiadm.