Naptastic

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====SR-IOV====
To make VFs exist, put a number <= NUM_OF_VFS into sriov_numvfs for that device. Before doing so, I recommend turning off VF probing. Otherwise the VFs will all make IPoIB interfaces, which probably isn't what you want. This setting is per PF.

I'm still checking if there's a way to configure the driver so this becomes the default setting.

  # echo 0 > /sys/class/infiniband/ibp13s0f0/device/sriov_drivers_autoprobe
  # echo 0 > /sys/class/infiniband/ibp13s0f1/device/sriov_drivers_autoprobe

If it works, there will be new PCI devices as well as VFs listed under `ip link`:

  # echo 7 > /sys/class/infiniband/ibp6s0f0/device/sriov_numvfs # no output on success; check dmesg for interesting but probably useless messages.
  # lspci | grep nfi
  06:00.0 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB]
  06:00.1 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB]
  06:00.2 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]
  06:00.3 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]
  06:00.4 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]
  06:00.5 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]
  06:00.6 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]
  06:00.7 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]
  06:01.0 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]

Warning: The output from ip link is very wide.

  # ip link
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
      link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
      link/ether 3c:ec:ef:6d:10:62 brd ff:ff:ff:ff:ff:ff
  3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
      link/ether 3c:ec:ef:6d:10:63 brd ff:ff:ff:ff:ff:ff
  4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 qdisc fq_codel state UP mode DEFAULT group default qlen 256
      link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
      vf 0     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
      vf 1     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
      vf 2     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
      vf 3     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
      vf 4     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
      vf 5     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
      vf 6     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
  5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc fq_codel state DOWN mode DEFAULT group default qlen 256
      link/infiniband 80:00:00:28:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:09 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

===VF Configuration===
The official documentation covers a sysfs interface for configuring VF properties. That interface hasn't existed for years. Before using a VF, you must set node_guid, port_guid, and state using ip link. Make port_guid == node_guid == unique. (I use the base port guid + VF + 1.)

GUIDs need to be set before attaching a VF to a VM. It should be possible to change state (simulating unplugging the cable) while a VM is using a VF but I haven't tested this.

  # ip link set dev ib0 vf 0 node_guid 58:49:56:0e:59:15:03:02
  # ip link set dev ib0 vf 0 port_guid 58:49:56:0e:59:15:03:02
  # ip link set dev ib0 vf 0 state enable

I keep the whole configuration in /etc/rc.local.

====GUIDs====
I think all of my Connect-IB cards have custom GUIDs. I'm not going to the trouble of installing and inspecting all of them right now.

  * 58:49:56:0e:53:66:01:01 - duckling Connect-IB
  * 58:49:56:0e:53:b7:0b:01 - not installed
  * 58:49:56:0e:59:15:03:01 - not installed
  * 7c:fe:90:03:00:a0:a0:80 - not installed
  * (there are several more uninstalled Connect-IB cards whose GUIDs I don't have written down.)