A few months ago I had to move suddenly and put my lab into storage. Where I moved, there was basic WiFi, and nowhere to set up a desktop. My web services were offline for weeks and I got pretty discouraged. Now I've got an opportunity to set it all up again, and enough people have expressed interest, I'm going to document and publish the whole process, or try anyway.

The first set of videos is going to be details on how my SAN is set up, along with a comparison of some of the things I've tried. The format consists of a description of each technology, when I do and don't use it and why, and then a little bit of actual how-to in case that technology appeals to you. I hope any instruction I provide is helpful.

You will almost certainly never see ATA over Ethernet used in production. It was used in a few SAN products but eventually lost out to iSCSI and Fibre Channel. I'm covering it anyway, and first mainly because it's a good teaching tool. It's easy to get started, and easy to show off different concepts that will become relevant with the more popular technologies. It's also a really handy tool to have in your toolbox for moving data.

For full support (initiator and target) you just need two packages:

apt install vblade aoetools

To export a block device to the network, you use a program called vblade. A daemonized version, vbladed, works with the same options. It starts a server that listens on layer 2 for ATA commands and responds to them. Here is (basically) how you use vblade:

vbladed shelf slot ethdev filename

Other options include sharing only part of a file, SYNC and DIRECT I/O modes, and buffer count. I/O modes and buffer counts require testing. Partial file sharing is there so the operator can logically divide a disk or file but in my opinion that's a bad enough idea I'm not even going to try it. Splitting a device for export is a concern that belongs to a filesystem, or a controller, or something that provides thin provisioning behind a strong layer of abstraction, like… a filesystem.

On BTRFS, if a directory has the +C attribute, you can preallocate a file of a given size and (is it contiguous?) it's pretty close to native I/O. (How close?)

ATA over Ethernet organizes disks by “slots” in “shelves”. The operator supplies the values.

• Shelf can be any value from 0-65534 except 4095.
• Slot can be any value from 0-254.
• Ethdev must be an Ethernet device. Bridges, VLAN interfaces, and VXLAN tunnels are all as good as gigabit Ethernet.

Unfortunately, vblade doesn't protect you from setting invalid values. If an invalid value is set, the initiator/client machine will get confused, probably fail to read the drive, and maybe give you a helpful error message, but probably not. (“Check DIP switches”?)

So, if you want to export a raw VM image from your current directory, you'd do this:

vbladed 1 1 eth0 vm.raw

…then on the initiator machine, run aoe-discover, aoe-stat, ls -al /dev/etherd

Once the remote device is in /dev, you can use it almost like any other block device. If it has partitions, Linux will find them automatically.

The one downside is that, for reasons I haven't investigated, AoE devices can't be attached directly to virtual machines. I wonder if KVM/QEMU doesn't like the device name; if that's the case, could udev rename the block device something more consistent with modern sensibilities? (E.g., /dev/aoe${shelf}s${slot}p${partition} and the partition indicator is optional?)

Difference between server/client and target/initiator:

1. server/client connections are ephemeral, but targets are expected never to go away. (Demo confusing aoe-discover)
2. target/initiator connections are mugh higher-performance, but typically require more configuration, and sacrifice flexibility.
3. storage protocols require guarantees about reliability, ordering, latency, and well-defined behavior if those guarantees aren't met.
4. Most importantly: target/initiator connections are only for storage and retrieval of data. The data that's returned from a target will always be whatever the initator stored there most recently.

ATA over Ethernet supports multipath natively and automatically. If AoE discovers a new link to the same slot and shelf on a different Ethernet interface, it will start sending commands and responses on both links round-robin, providing both failover and load balancing. In my experience, speed increases almost linearly with the number of links added. My personal best over GigE is 519 MiB/s using five links. Using VXLAN over InfiniBand, I got 1.4 GiB/s, but that was probably limited by the drive. Benchmarks later.

Section 1.1 of the ATA over Ethernet standard: “AoE is not a connection based protocol. Each message sent to a server should be considered unique and unreliable.”

https://web.archive.org/web/20161025044402/http://brantleycoilecompany.com/AoEr11.pdf

When something goes wrong such as a link disappearing, AoE blocks for 10 seconds by default. That's a long time for your users to be wondering what's going on, and it only has to happen a couple of times before they stop trusting you. The timeout value lives in and is specified in seconds. A shorter value would make more sense. In the context of a modern SAN, 10ms is plenty for a timeout. Maybe it should be higher for spindle drives–whatever, I can't change the kernel. At least I don't think I can.

vblade and vbladed do not maintain state between or across instances. If you need an ATA over Ethernet export to come back after a reboot, you will need your OS to manage vblade processes. On Debian, that's done by putting shell script fragments in /etc/vblade.conf.d/. There is an example file installed there with all directives commented out:

# This is a POSIX shell fragment

# configuration of a single vblade instance

# Supported variables:

# shelf address. Mandatory
# shelf=

# slot address. Mandatory
# slot=

# Network interface name. Mandatory
# netif=

# The name of the regular file or block device to export. Mandatory
# filename=

# Other options, see vblade(8)
# options=

# ionice=
# Set the I/O scheduling class and priority.
# Must be understood by ionice(1)

# Example:
# shelf=10
# slot=3
# netif=em3
# filename=/dev/mapper/export
# options='-m 11:22:33:44:55:66 -o 8'
# ionice='--class best-effort --classdata 7'

ATA over Ethernet is intended to run inside of trusted networks. By default, it runs wide open: any host in the same layer 2 broadcast domain has full access to any exported volume. There is no distinction between read-only and read-write access. Preventing unwanted access has to be done by dividing broadcast domains. Originally that meant physical separation–different network adapters, cables, and switches. Now, that separation is more likely to be implemented inside the switch using VLANs or VXLAN tunnels.

SAN technologies generally have some kind of ACL mechanism. This has benefits for security and discoverability. As a configuration or command-line option, vblade can take one or more MAC addresses to which to restrict access. Hosts not on the list can't (see|access) that device. This should not be considered an especially robust mechanism since Ethernet addresses are nearly trivial to spoof.

As you put these values into these configuration files, imagine that you are actually plugging different hard drives into different computers. It's not about moving data to a different drive anymore; it's about moving the drive to where the user needs it to be, and doing so in a completely virtual way.

And that brings us neatly to maybe the most useful thing about a SAN: It makes local storage unnecessary. iPXE supports ATA over Ethernet directly. The DHCP has to provide a suitable root-path option. For isc-dhcp-server, telling a host to boot from shelf 12, slot 9 looks like this:

option root-path "aoe:e12.9";

The DHCP server must not also provide a TFTP next-server and filename. If it does, iPXE will boot via TFTP instead.

As far as I can tell, there's no way to have your root volume on ATA over Ethernet. iPXE can use AoE to fetch a bootloader, but that's it: neither Linux nor Windows can use it as a root volume.