Naptastic

A few months ago I had to move suddenly and put my lab into storage. Where I moved, there was basic WiFi, and nowhere to set up a desktop. My web services were offline for weeks and I got pretty discouraged. Now I've got an opportunity to set it all up again, and enough people have expressed interest, I'm going to document and publish the whole process, or try anyway.

Follow-through is not my forte though, so I'm giving myself an incentive–an ulterior motive, if you will: I really want some faster network gear and equipment that can do SR-IOV so that I can play with hyperconvergence. If I finish this project, I'm going to buy myself that gear (unless somebody else buys it for me.) Even though I really can't justify the expense, space used, or power consumed, if I can prove to myself that I'm capable of finishing a project like this, then dammit I have earned my shinies. And I will use them to make videos about the cool things you can do with the faster networking equipment, software-defined networking and all that.

The first set of videos is going to be details on how my SAN is set up, along with a comparison of all the things I've tried. The format consists of a description of each technology, when I do and don't use it and why, and then a little bit of actual how-to in case that technology appeals to you. I hope any instruction I provide is helpful.

You will almost certainly never see ATA over Ethernet used in production. It was used in a few SAN products but eventually lost out to iSCSI and Fibre Channel. I'm covering it anyway, and first mainly because it's a good teaching tool. It's easy to get started, and easy to show off different concepts that will become relevant with the more popular technologies. It's also a really handy tool to have in your toolbox for moving data.

Right now, it has a bug that can cause systems on the network not to shut down or reboot if there's an AoE server on the network, so it shouldn't be used in production. (I need to dig into this.) You can avoid the problem by disconnecting all AoE devices and unloading the aoe module before shutting down the host:

rmmod aoe

If my testing is right, the only things necessary for a host to crash on shutdown are (1) there is an ATA-over-Ethernet device in a broadcast domain your host is part of, and (2) the aoe module is loaded.

For full support (initiator and target) you just need two packages:

apt install vblade aoetools

To export a block device to the network, you use a program called vblade. A daemonized version, vbladed, works with the same options. It starts a server that listens on layer 2 for ATA commands and responds to them. Here is (basically) how you use vblade:

vbladed shelf slot ethdev filename

Other options include sharing only part of a file, SYNC and DIRECT I/O modes, and buffer count. I/O modes and buffer counts require testing. Partial file sharing is there so the operator can logically divide a disk or file but in my opinion that's a bad enough idea I'm not even going to try it. Splitting a device for export is a concern that belongs to a filesystem, or a controller, or something that provides thin provisioning behind a strong layer of abstraction, like… a filesystem.

On BTRFS, if a directory has the +C attribute, you can preallocate a file of a given size and (is it contiguous?) it's pretty close to native I/O. (How close?)

ATA over Ethernet organizes disks by “slots” in “shelves”. The operator supplies the values.

• Shelf can be any value from 0-65534 except 4095.
• Slot can be any value from 0-254.
• Ethdev must be an Ethernet device. Bridges, VLAN interfaces, and VXLAN tunnels are all as good as gigabit Ethernet.

Unfortunately, vblade doesn't protect you from setting invalid values. If an invalid value is set, the initiator/client machine will get confused, probably fail to read the drive, and maybe give you a helpful error message, but probably not. (“Check DIP switches”?)

So, if you want to export a raw VM image from your current directory, you'd do this:

vbladed 1 1 eth0 vm.raw

…then on the initiator machine, run aoe-discover, aoe-stat, ls -al /dev/etherd

Once the remote device is in /dev, you can use it almost like any other block device. If it has partitions, Linux will find them automatically.

The one downside is that, for reasons I haven't investigated, AoE devices can't be attached directly to virtual machines. I wonder if KVM/QEMU doesn't like the device name; if that's the case, could udev rename the block device something more consistent with modern sensibilities? (E.g., /dev/aoe${shelf}s${slot}p${partition} and the partition indicator is optional?)

Difference between server/client and target/initiator:

1. server/client connections are ephemeral, but targets are expected never to go away. (Demo confusing aoe-discover)
2. target/initiator connections are mugh higher-performance, but typically require more configuration, and sacrifice flexibility.
3. storage protocols require guarantees about reliability, ordering, latency, and well-defined behavior if those guarantees aren't met.
4. Most importantly: target/initiator connections are only for storage and retrieval of data. The data that's returned from a target will always be whatever the initator stored there most recently.

ATA over Ethernet supports multipath natively and automatically. If AoE discovers a new link to the same slot and shelf on a different Ethernet interface, it will start sending commands and responses on both links round-robin, providing both failover and load balancing. In my experience, speed increases almost linearly with the number of links added. My personal best over GigE is 519 MiB/s using five links. Using VXLAN over InfiniBand, I got 1.4 GiB/s, but that was probably limited by the drive. Benchmarks later.

Section 1.1 of the ATA over Ethernet standard: “AoE is not a connection based protocol. Each message sent to a server should be considered unique and unreliable.”

https://web.archive.org/web/20161025044402/http://brantleycoilecompany.com/AoEr11.pdf

When something goes wrong such as a link disappearing, AoE blocks for 10 seconds by default. That's a long time for your users to be wondering what's going on, and it only has to happen a couple of times before they stop trusting you. The timeout value lives in and is specified in seconds. A shorter value would make more sense. In the context of a modern SAN, 10ms is plenty for a timeout. Maybe it should be higher for spindle drives–whatever, I can't change the kernel. At least I don't think I can.

vblade and vbladed do not maintain state between or across instances. If you need an ATA over Ethernet export to come back after a reboot, you will need your OS to manage vblade processes. On Debian, that's done by putting shell script fragments in /etc/vblade.conf.d/. There is an example file installed there with all directives commented out:

# This is a POSIX shell fragment

# configuration of a single vblade instance

# Supported variables:

# shelf address. Mandatory
# shelf=

# slot address. Mandatory
# slot=

# Network interface name. Mandatory
# netif=

# The name of the regular file or block device to export. Mandatory
# filename=

# Other options, see vblade(8)
# options=

# ionice=
# Set the I/O scheduling class and priority.
# Must be understood by ionice(1)

# Example:
# shelf=10
# slot=3
# netif=em3
# filename=/dev/mapper/export
# options='-m 11:22:33:44:55:66 -o 8'
# ionice='--class best-effort --classdata 7'

ATA over Ethernet is intended to run inside of trusted networks. By default, it runs wide open: any host in the same layer 2 broadcast domain can access Originally that meant physical separation–different switches. Now that separation is more likely to be implemented inside the switch using VLANs or VXLAN tunnels. [show off VLAN setup iterations. Does performance change? Testing needed…]

SAN technologies generally have some kind of ACL mechanism. This has benefits for security and discoverability. As a configuration or command-line option, vblade can take one or more MAC addresses to which to restrict access. Hosts not on the list can't (see|access) that device. This should not be considered an especially robust mechanism since Ethernet addresses are nearly trivial to spoof.

need examples of config file options for ACLs.

As you put these values into these configuration files, imagine that you are actually plugging different hard drives into different computers. It's not about moving data to a different drive anymore; it's about moving the drive to where the user needs it to be, and doing so in a completely virtual way.

And that brings us neatly to maybe the most useful thing about a SAN: It makes local storage unnecessary. iPXE supports ATA over Ethernet natively; the DHCP server just has to provide options