Naptastic

To make VFs exist, put a number ⇐ NUM_OF_VFS into sriov_numvfs for that device. Before doing so, I recommend turning off VF probing. Otherwise the VFs will all make IPoIB interfaces, which probably isn't what you want. This setting is per PF.

I'm still checking if there's a way to configure the driver so this becomes the default setting.

# echo 0 > /sys/class/infiniband/ibp13s0f0/device/sriov_drivers_autoprobe
# echo 0 > /sys/class/infiniband/ibp13s0f1/device/sriov_drivers_autoprobe

If it works, there will be new PCI devices as well as VFs listed under `ip link`:

# echo 7 > /sys/class/infiniband/ibp6s0f0/device/sriov_numvfs # no output on success; check dmesg for interesting but probably useless messages.
# lspci | grep nfi
06:00.0 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB]
06:00.1 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB]
06:00.2 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]
06:00.3 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]
06:00.4 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]
06:00.5 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]
06:00.6 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]
06:00.7 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]
06:01.0 Infiniband controller: Mellanox Technologies MT27600 Family [Connect-IB Virtual Function]

Warning: The output from ip link is very wide.

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 3c:ec:ef:6d:10:62 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 3c:ec:ef:6d:10:63 brd ff:ff:ff:ff:ff:ff
4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 4092 qdisc fq_codel state UP mode DEFAULT group default qlen 256
    link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    vf 0     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
    vf 1     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
    vf 2     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
    vf 3     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
    vf 4     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
    vf 5     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
    vf 6     link/infiniband 80:00:00:29:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:01 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff, spoof checking off, NODE_GUID 00:00:00:00:00:00:00:00, PORT_GUID 00:00:00:00:00:00:00:00, link-state disable, trust off, query_rss off
5: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc fq_codel state DOWN mode DEFAULT group default qlen 256
    link/infiniband 80:00:00:28:fe:80:00:00:00:00:00:00:58:49:56:0e:53:b7:0b:09 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

The official documentation covers a sysfs interface for configuring VF properties. That interface hasn't existed for years. Before using a VF, you must set node_guid, port_guid, and state using ip link. Make port_guid == node_guid == unique. (I use the base port guid + VF + 1.)

GUIDs need to be set before attaching a VF to a VM. It should be possible to change state (simulating unplugging the cable) while a VM is using a VF but I haven't tested this.

I keep the whole configuration in /etc/rc.local.