Naptastic

A few months ago I had to move suddenly and put my lab into storage. Where I moved, there was basic WiFi, and nowhere to set up a desktop. My web services were offline for weeks and I got pretty discouraged. Now I've got an opportunity to set it all up again, and enough people have expressed interest, I'm going to document and publish the whole process, or try anyway.

Follow-through is not my forte though, so I'm giving myself an incentive–an ulterior motive, if you will: I really want some faster network gear and equipment that can do SR-IOV so that I can play with hyperconvergence. If I finish this project, I'm going to buy myself that gear (unless somebody else buys it for me.) Even though I really can't justify the expense, space used, or power consumed, if I can prove to myself that I'm capable of finishing a project like this, then dammit I have earned my shinies. And I will use them to make videos about the cool things you can do with the faster networking equipment, software-defined networking and all that.

The first set of videos is going to be details on how my SAN is set up, along with a comparison of all the things I've tried. The format consists of a description of each technology, when I do and don't use it and why, and then a little bit of actual how-to in case that technology appeals to you. I hope any instruction I provide is helpful.

You will almost certainly never see ATA over Ethernet used in production. It was used in a few SAN products but eventually lost out to iSCSI and Fibre Channel. I'm covering it anyway, and first mainly because it's a good teaching tool. It's easy to get started, and easy to show off different concepts that will become relevant with the more popular technologies. It's a really handy tool to have in your toolbox for moving data if all you have is Ethernet.

Right now, it has a bug that can cause systems on the network not to shut down or reboot if there's an AoE server on the network, so it shouldn't be used in production. (I need to dig into this.)

To export a block device to the network, you use a program called vblade. A daemonized version, vbladed, works with the same options. It starts a server that listens on layer 2 for ATA commands and responds to them. Here is (basically) how you use vblade:

vbladed shelf slot ethdev filename

Other options include sharing only part of a file, SYNC and DIRECT I/O modes, and buffer count. I/O modes and buffer counts require testing. Partial file sharing is there so the operator can logically divide a disk or file but in my opinion that's a bad enough idea I'm not even going to try it. Splitting a device for export is a concern that belongs to a filesystem, or a controller, or something that provides thin provisioning behind a strong layer of abstraction, like… a filesystem.

On BTRFS, if a directory has the +C attribute, you can preallocate a file of a given size and (is it contiguous?) it's pretty close to native I/O. (How close?)

ATA over Ethernet organizes disks by “slots” in “shelves”. The operator supplies the values. Shelf can be any value from 0-65534 except 4095. Slot can be any value from 0-254. Unfortunately, vblade doesn't protect you from setting invalid values. If an invalid value is set, the initiator/client machine will get confused, probably fail to read the drive, and maybe give you a helpful error message, but probably not. (“Check DIP switches”?)

So, if you want to export a raw VM image from your current directory, you'd do this:

vbladed 1 1 eth0 vm.raw

…then on the initiator machine, run aoe-discover, aoe-stat, ls -al /dev/etherd

Once the remote device is in /dev, you can use it like any other device. If it has partitions, Linux will find them automatically. Attaching it to a virtual machine is especially handy, since it will have the same device name on any system that can access it. (Demo VM migration with AoE backing store?)

Difference between server/client and target/initiator:

1. server/client connections are ephemeral
2. targets are expected never to go away. (Confuse aoe-discover)
3. the paradigm is different
4. high performance
5. sacrifices in flexibility
6. requires low latency and delivery guarantees

Section 1.1 of the ATA over Ethernet standard: “AoE is not a connection based protocol. Each message sent to a server should be considered unique and unreliable.”

https://web.archive.org/web/20161025044402/http://brantleycoilecompany.com/AoEr11.pdf

When something goes wrong such as a link disappearing, AoE blocks for 10 seconds by default. That's a long time for your users to be wondering what's going on, and it only has to happen a couple of times before they stop trusting you.

So I glossed over security and VLANs earlier. ATA over Ethernet is designed to run inside of trusted networks. By default, it runs wide open: any host in the same layer 2 broadcast domain can access Originally that meant physical separation. Now that separation is more likely to be implemented with VLANs. [show off VLAN setup iterations. Does performance change? Testing needed…]

AoE can also restrict access by MAC address. MAC addresses are easy to spoof so this isn't actually secure. It's just good practice for what's coming. [lightning strike, lol?]

As you put these values into these configuration files, imagine that you are actually plugging different hard drives into different computers. It's not about moving data to a different drive anymore; it's about moving the drive to where the user needs it to be, and doing so in a completely virtual way.

→ ACLs → VLANs

ATA over Ethernet supports multipath natively and automatically. If AoE discovers a new link to the same device, it will start sending commands and responses on each link round-robin, providing both failover and load balancing. In my experience, speed increases almost linearly with the number of links added. My personal best is 519 MiB/s using five gigabit Ethernet links, but I might be able to do better than that. Benchmarks later.