TODO Please put storage traffic on its own IB subnet and limit things sanely.

The goal is to run everything over RDMA (InfiniBand specifically) and maybe have TCP as a fallback. (If TCP, then I probably want to try VMA... which I don't want to do.)

====Target====
NVMe calls targets "subsystems". Good for them.

Create the subsystem:
    /> subsystems/ create nqn.2014-08.rocks.narf.southpark

Create a port and set its properties:
    /> ports/ create 1
    /> ports/1/ set addr trtype=rdma
    /> ports/1/ set addr adrfam=ipv4
    /> ports/1/ set addr traddr=172.20.64.13
    /> ports/1/ set addr trsvcid=4420
    /> ports/1/subsystems create nqn.2014-08.rocks.narf.southpark 

NVMe uses "namespaces" (which are numbers) instead of LUNs. Good for them.

Create, set, and enable namespace:
    /> subsystems/nqn.2014-08.rocks.narf.southpark/namespaces create 1
    /> subsystems/nqn.2014-08.rocks.narf.southpark/namespaces/1 set device path=/dev/nvme0n1
    /> subsystems/nqn.2014-08.rocks.narf.southpark/namespaces/1 enable

You can use a file as a backstore. The syntax is not at all obvious. Why "group=device"? Whatever:
    />subsystems/nqn.2014-08.rocks.narf.hostname/namespaces/1 set group=device path=/path/to/some/host.img

I haven't checked if this works for partitions or SATA drives, but I think it should?

I haven't figured out ramdisks yet. There isn't a built-in system like in LIO.

Create ACLs:
    /> hosts/ create nqn.2014-08.rocks.narf.sadness
    /> subsystems/nqn.2014-08.rocks.narf.southpark/ set attr allow_any_host=0
    Parameter allow_any_host is now '0'.
    /> subsystems/nqn.2014-08.rocks.narf.southpark/allowed_hosts create nqn.2014-08.rocks.narf.sadness 

====Initiator====
NVMe calls initiators "hosts". Good for them.

Load the module (should be able to list this in /etc/modules)
    # modprobe nvme-rdma

Discovery:
    # nvme discover -t rdma -a 172.20.64.13 -s 4420

Log in:
    # nvme connect -t rdma -n nqn.2014-08.rocks.narf.southpark -a 172.20.64.13 -s 4420

Rescan:
    # nvme ns-rescan /dev/nvme1

  * If a namespace you expect to appear isn't appearing, check the target to see if you forget to enable it

Disconnect a subsystem:
    # nvme disconnect -d /dev/nvme1

Gotta admit, that feels a lot nicer than logging out of iscsiadm.