Naptastic

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
TODO Please put storage traffic on its own IB subnet and limit things sanely.

The goal is to run everything over RDMA (InfiniBand specifically) and maybe have TCP as a fallback. (If TCP, then I probably want to try VMA... which I don't want to do.)

====Target====
NVMe calls targets "subsystems". Good for them.

Create the subsystem:
    /> subsystems/ create nqn.2014-08.rocks.narf.southpark

Create a port and set its properties:
    /> ports/ create 1
    /> ports/1/ set addr trtype=rdma
    /> ports/1/ set addr adrfam=ipv4
    /> ports/1/ set addr traddr=172.20.64.13
    /> ports/1/ set addr trsvcid=4420
    /> ports/1/subsystems create nqn.2014-08.rocks.narf.southpark 

NVMe uses "namespaces" (which are numbers) instead of LUNs. Good for them.

Create, set, and enable namespace:
    /> subsystems/nqn.2014-08.rocks.narf.southpark/namespaces create 1
    /> subsystems/nqn.2014-08.rocks.narf.southpark/namespaces/1 set device path=/dev/nvme0n1
    /> subsystems/nqn.2014-08.rocks.narf.southpark/namespaces/1 enable

You can use a file as a backstore. The syntax is not at all obvious. Why "group=device"? Whatever:
    />subsystems/nqn.2014-08.rocks.narf.hostname/namespaces/1 set group=device path=/path/to/some/host.img

I haven't checked if this works for partitions or SATA drives, but I think it should?

I haven't figured out ramdisks yet. There isn't a built-in system like in LIO.

Create ACLs:
    /> hosts/ create nqn.2014-08.rocks.narf.sadness
    /> subsystems/nqn.2014-08.rocks.narf.southpark/ set attr allow_any_host=0
    Parameter allow_any_host is now '0'.
    /> subsystems/nqn.2014-08.rocks.narf.southpark/allowed_hosts create nqn.2014-08.rocks.narf.sadness 

====Initiator====
NVMe calls initiators "hosts". Good for them.

Load the module (should be able to list this in /etc/modules)
    # modprobe nvme-rdma

Discovery:
    # nvme discover -t rdma -a 172.20.64.13 -s 4420

Log in:
    # nvme connect -t rdma -n nqn.2014-08.rocks.narf.southpark -a 172.20.64.13 -s 4420

Rescan:
    # nvme ns-rescan /dev/nvme1

  * If a namespace you expect to appear isn't appearing, check the target to see if you forget to enable it

Disconnect a subsystem:
    # nvme disconnect -d /dev/nvme1

Gotta admit, that feels a lot nicer than logging out of iscsiadm.