Differences

This shows you the differences between two versions of the page.

--- nndocs:ata-over-ethernet [2024/03/10 20:17] – [Perfect vs. Good: A Fight to the Death] add info about the problem with using aoe naptastic
+++ nndocs:ata-over-ethernet [2024/08/23 16:02] (current) – remove bug; no longer able to reproduce. naptastic
@@ Line 3: / Line 3: @@
 ====Perfect vs. Good: A Fight to the Death====
-----
 ===Preface (maybe doesn't belong in this video?)===
 A few months ago I had to move suddenly and put my lab into storage. Where I moved, there was basic WiFi, and nowhere to set up a desktop. My web services were offline for weeks and I got pretty discouraged. Now I've got an opportunity to set it all up again, and enough people have expressed interest, I'm going to document and publish the whole process, or try anyway.
-Follow-through is not my forte though, so I'm giving myself an incentive--an ulterior motive, if you will: I really want some faster network gear and equipment that can do SR-IOV so that I can play with hyperconvergence. If I finish this project, I'm going to buy myself that gear (unless somebody else buys it for me.) Even though I really can't justify the expense, space used, or power consumed, if I can prove to myself that I'm capable of finishing a project like this, then dammit I have earned my shinies. And I will use them to make videos about the cool things you can do with the faster networking equipment, software-defined networking and all that.
+The first set of videos is going to be details on how my SAN is set up, along with a comparison of some of the things I've tried. The format consists of a description of each technology, when I do and don't use it and why, and then a little bit of actual how-to in case that technology appeals to you. I hope any instruction I provide is helpful.
-The first set of videos is going to be details on how my SAN is set up, along with a comparison of all the things I've tried. The format consists of a description of each technology, when I do and don't use it and why, and then a little bit of actual how-to in case that technology appeals to you. I hope any instruction I provide is helpful.
+===Introduction to ATA over Ethernet (AoE)===
+You will almost certainly never see ATA over Ethernet used in production. It was used in a few SAN products but eventually lost out to iSCSI and Fibre Channel. I'm covering it anyway, and first mainly because it's a good teaching tool. It's easy to get started, and easy to show off different concepts that will become relevant with the more popular technologies. It's also a really handy tool to have in your toolbox for moving data.
-----
-You will almost certainly never see ATA over Ethernet used in production. It was used in a few SAN products but eventually lost out to iSCSI and Fibre Channel. I'm covering it anyway, and first mainly because it's a good teaching tool. It's easy to get started, and easy to show off different concepts that will become relevant with the more popular technologies. It's a really handy tool to have in your toolbox for moving data if all you have is Ethernet.
-Right now, it has a bug that can cause systems on the network not to shut down or reboot if there's an AoE server on the network, so it shouldn't be used in production. (I need to dig into this.) You can avoid the problem by disconnecting all AoE devices and unloading the aoe module before shutting down the host:
-    rmmod aoe
-If my testing is right, the only things necessary for a host to crash on shutdown are (1) there is an ATA-over-Ethernet device in a broadcast domain your host is part of, and (2) the aoe module is loaded.
 For full support (initiator and target) you just need two packages:
+  apt install vblade aoetools
-    apt install vblade aoetools
 To export a block device to the network, you use a program called vblade. A daemonized version, vbladed, works with the same options. It starts a server that listens on layer 2 for ATA commands and responds to them. Here is (basically) how you use vblade:
+  vbladed shelf slot ethdev filename
-    vbladed shelf slot ethdev filename
 Other options include sharing only part of a file, SYNC and DIRECT I/O modes, and buffer count. I/O modes and buffer counts require testing. Partial file sharing is there so the operator can logically divide a disk or file but in my opinion that's a bad enough idea I'm not even going to try it. Splitting a device for export is a concern that belongs to a filesystem, or a controller, or something that provides thin provisioning behind a strong layer of abstraction, like... a filesystem.
@@ Line 34: / Line 22: @@
 On BTRFS, if a directory has the +C attribute, you can preallocate a file of a given size and (is it contiguous?) it's pretty close to native I/O. (How close?)
-ATA over Ethernet organizes disks by "slots" in "shelves". The operator supplies the values. Shelf can be any value from 0-65534 except 4095. Slot can be any value from 0-254. Unfortunately, vblade doesn't protect you from setting invalid values. If an invalid value is set, the initiator/client machine will get confused, probably fail to read the drive, and maybe give you a helpful error message, but probably not. ("Check DIP switches"?)
+ATA over Ethernet organizes disks by "slots" in "shelves". The operator supplies the values.
+  * Shelf can be any value from 0-65534 except 4095.
+  * Slot can be any value from 0-254.
+  * Ethdev must be an **Ethernet** device. Bridges, VLAN interfaces, and VXLAN tunnels are all as good as gigabit Ethernet.
+Unfortunately, vblade doesn't protect you from setting invalid values. If an invalid value is set, the initiator/client machine will get confused, probably fail to read the drive, and maybe give you a helpful error message, but probably not. ("Check DIP switches"?)
 So, if you want to export a raw VM image from your current directory, you'd do this:
+  vbladed 1 1 eth0 vm.raw
-    vbladed 1 1 eth0 vm.raw
 ...then on the initiator machine, run aoe-discover, aoe-stat, ls -al /dev/etherd
-Once the remote device is in /dev, you can use it like any other device. If it has partitions, Linux will find them automatically. Attaching it to a virtual machine is especially handy, since it will have the same device name on any system that can access it. (Demo VM migration with AoE backing store?)
+Once the remote device is in /dev, you can use it almost like any other block device. If it has partitions, Linux will find them automatically.
-----
+The one downside is that, for reasons I haven't investigated, AoE devices can't be attached directly to virtual machines. I wonder if KVM/QEMU doesn't like the device name; if that's the case, could udev rename the block device something more consistent with modern sensibilities? (E.g., /dev/aoe${shelf}s${slot}p${partition} and the partition indicator is optional?)
+===Aside: Why "Target" and "Initiator"?===
 Difference between server/client and target/initiator:
   - server/client connections are ephemeral, but targets are expected never to go away. (Demo confusing aoe-discover)
-  - high performance, but requires more configuration, and sacrifices flexibility
+  - target/initiator connections are mugh higher-performance, but typically require more configuration, and sacrifice flexibility.
-  - requires low latency and delivery guarantees
+  - storage protocols require guarantees about reliability, ordering, latency, and well-defined behavior if those guarantees aren't met.
-  - Most importantly: target/initiator connections are **only** for storage and retrieval of data.
+  - Most importantly: target/initiator connections are **only** for storage and retrieval of data. The data that's returned from a target will always be whatever the initator stored there most recently.
+===Multipath===
+ATA over Ethernet supports multipath natively and automatically. If AoE discovers a new link to the same slot and shelf on a different Ethernet interface, it will start sending commands and responses on both links round-robin, providing both failover and load balancing. In my experience, speed increases almost linearly with the number of links added. My personal best over GigE is 519 MiB/s using five links. Using VXLAN over InfiniBand, I got 1.4 GiB/s, but that was probably limited by the drive. Benchmarks later.
 Section 1.1 of the ATA over Ethernet standard:
@@ Line 57: / Line 53: @@
 https://web.archive.org/web/20161025044402/http://brantleycoilecompany.com/AoEr11.pdf
-When something goes wrong such as a link disappearing, AoE blocks for 10 seconds by default. That's a long time for your users to be wondering what's going on, and it only has to happen a couple of times before they stop trusting you.
+When something goes wrong such as a link disappearing, AoE blocks for 10 seconds by default. That's a long time for your users to be wondering what's going on, and it only has to happen a couple of times before they stop trusting you. The timeout value lives in FIXME and is specified in seconds. A shorter value would make more sense. In the context of a modern SAN, 10ms is **plenty** for a timeout. Maybe it should be higher for spindle drives--whatever, I can't change the kernel. At least I don't think I can.
-----
+===Persistent Configuration===
+vblade and vbladed do not maintain state between or across instances. If you need an ATA over Ethernet export to come back after a reboot, you will need your OS to manage vblade processes. On Debian, that's done by putting shell script fragments in /etc/vblade.conf.d/. There is an example file installed there with all directives commented out:
-So I glossed over security and VLANs earlier. ATA over Ethernet is designed to run inside of trusted networks. By default, it runs wide open: any host in the same layer 2 broadcast domain can access Originally that meant physical separation. Now that separation is more likely to be implemented with VLANs. [show off VLAN setup iterations. Does performance change? Testing needed...]
+  # This is a POSIX shell fragment
+  # configuration of a single vblade instance
+  # Supported variables:
+  # shelf address. Mandatory
+  # shelf=
+  # slot address. Mandatory
+  # slot=
+  # Network interface name. Mandatory
+  # netif=
+  # The name of the regular file or block device to export. Mandatory
+  # filename=
+  # Other options, see vblade(8)
+  # options=
+  # ionice=
+  # Set the I/O scheduling class and priority.
+  # Must be understood by ionice(1)
+  # Example:
+  # shelf=10
+  # slot=3
+  # netif=em3
+  # filename=/dev/mapper/export
+  # options='-m 11:22:33:44:55:66 -o 8'
+  # ionice='--class best-effort --classdata 7'
-----
+===Security===
+ATA over Ethernet is intended to run inside of trusted networks. By default, it runs wide open: any host in the same layer 2 broadcast domain has full access to any exported volume. There is no distinction between read-only and read-write access. Preventing unwanted access has to be done by dividing broadcast domains. Originally that meant physical separation--different network adapters, cables, and switches. Now, that separation is more likely to be implemented inside the switch using VLANs or VXLAN tunnels.
-AoE can also restrict access by MAC address. MAC addresses are easy to spoof so this isn't actually secure. It's just good practice for what's coming. [lightning strike, lol?]
+SAN technologies generally have some kind of ACL mechanism. This has benefits for security and discoverability. As a configuration or command-line option, vblade can take one or more MAC addresses to which to restrict access. Hosts not on the list can't (see|access) that device. This should not be considered an especially robust mechanism since Ethernet addresses are nearly trivial to spoof.
 As you put these values into these configuration files, imagine that you are actually plugging different hard drives into different computers. It's not about moving data to a different drive anymore; it's about moving the drive to where the user needs it to be, and doing so in a completely virtual way.
--> ACLs
+===Boot===
--> VLANs
+And that brings us neatly to maybe the most useful thing about a SAN: It makes local storage unnecessary. iPXE supports ATA over Ethernet directly. The DHCP has to provide a suitable root-path option. For isc-dhcp-server, telling a host to boot from shelf 12, slot 9 looks like this:
-----
+  option root-path "aoe:e12.9";
-ATA over Ethernet supports multipath natively and automatically. If AoE discovers a new link to the same device, it will start sending commands and responses on each link round-robin, providing both failover and load balancing. In my experience, speed increases almost linearly with the number of links added. My personal best is 519 MiB/s using five gigabit Ethernet links, but I might be able to do better than that. Benchmarks later.
+The DHCP server must not also provide a TFTP next-server and filename. If it does, iPXE will boot via TFTP instead.
+FIXME As far as I can tell, there's no way to have your root volume on ATA over Ethernet. iPXE can use AoE to fetch a bootloader, but that's it: neither Linux nor Windows can use it as a root volume.