Differences

This shows you the differences between two versions of the page.

--- nndocs:ata-over-ethernet [2024/03/15 17:04] – The whole outline is there now naptastic
+++ nndocs:ata-over-ethernet [2024/08/23 16:02] (current) – remove bug; no longer able to reproduce. naptastic
@@ Line 3: / Line 3: @@
 ====Perfect vs. Good: A Fight to the Death====
-----
 ===Preface (maybe doesn't belong in this video?)===
 A few months ago I had to move suddenly and put my lab into storage. Where I moved, there was basic WiFi, and nowhere to set up a desktop. My web services were offline for weeks and I got pretty discouraged. Now I've got an opportunity to set it all up again, and enough people have expressed interest, I'm going to document and publish the whole process, or try anyway.
-Follow-through is not my forte though, so I'm giving myself an incentive--an ulterior motive, if you will: I really want some faster network gear and equipment that can do SR-IOV so that I can play with hyperconvergence. If I finish this project, I'm going to buy myself that gear (unless somebody else buys it for me.) Even though I really can't justify the expense, space used, or power consumed, if I can prove to myself that I'm capable of finishing a project like this, then dammit I have earned my shinies. And I will use them to make videos about the cool things you can do with the faster networking equipment, software-defined networking and all that.
+The first set of videos is going to be details on how my SAN is set up, along with a comparison of some of the things I've tried. The format consists of a description of each technology, when I do and don't use it and why, and then a little bit of actual how-to in case that technology appeals to you. I hope any instruction I provide is helpful.
-The first set of videos is going to be details on how my SAN is set up, along with a comparison of all the things I've tried. The format consists of a description of each technology, when I do and don't use it and why, and then a little bit of actual how-to in case that technology appeals to you. I hope any instruction I provide is helpful.
-----
 ===Introduction to ATA over Ethernet (AoE)===
 You will almost certainly never see ATA over Ethernet used in production. It was used in a few SAN products but eventually lost out to iSCSI and Fibre Channel. I'm covering it anyway, and first mainly because it's a good teaching tool. It's easy to get started, and easy to show off different concepts that will become relevant with the more popular technologies. It's also a really handy tool to have in your toolbox for moving data.
-Right now, it has a bug that can cause systems on the network not to shut down or reboot if there's an AoE server on the network, so it shouldn't be used in production. (I need to dig into this.) You can avoid the problem by disconnecting all AoE devices and unloading the aoe module before shutting down the host:
-  rmmod aoe
-If my testing is right, the only things necessary for a host to crash on shutdown are (1) there is an ATA-over-Ethernet device in a broadcast domain your host is part of, and (2) the aoe module is loaded.
 For full support (initiator and target) you just need two packages:
@@ Line 65: / Line 56: @@
 ===Persistent Configuration===
-FIXME
+vblade and vbladed do not maintain state between or across instances. If you need an ATA over Ethernet export to come back after a reboot, you will need your OS to manage vblade processes. On Debian, that's done by putting shell script fragments in /etc/vblade.conf.d/. There is an example file installed there with all directives commented out:
+  # This is a POSIX shell fragment
+  # configuration of a single vblade instance
+  # Supported variables:
+  # shelf address. Mandatory
+  # shelf=
+  # slot address. Mandatory
+  # slot=
+  # Network interface name. Mandatory
+  # netif=
+  # The name of the regular file or block device to export. Mandatory
+  # filename=
+  # Other options, see vblade(8)
+  # options=
+  # ionice=
+  # Set the I/O scheduling class and priority.
+  # Must be understood by ionice(1)
+  # Example:
+  # shelf=10
+  # slot=3
+  # netif=em3
+  # filename=/dev/mapper/export
+  # options='-m 11:22:33:44:55:66 -o 8'
+  # ionice='--class best-effort --classdata 7'
 ===Security===
-So I glossed over security and VLANs earlier. ATA over Ethernet is designed to run inside of trusted networks. By default, it runs wide open: any host in the same layer 2 broadcast domain can access Originally that meant physical separation--different switches. Now that separation is more likely to be implemented inside the switch using VLANs or VXLAN tunnels. [show off VLAN setup iterations. Does performance change? Testing needed...]
+ATA over Ethernet is intended to run inside of trusted networks. By default, it runs wide open: any host in the same layer 2 broadcast domain has full access to any exported volume. There is no distinction between read-only and read-write access. Preventing unwanted access has to be done by dividing broadcast domains. Originally that meant physical separation--different network adapters, cables, and switches. Now, that separation is more likely to be implemented inside the switch using VLANs or VXLAN tunnels.
 SAN technologies generally have some kind of ACL mechanism. This has benefits for security and discoverability. As a configuration or command-line option, vblade can take one or more MAC addresses to which to restrict access. Hosts not on the list can't (see|access) that device. This should not be considered an especially robust mechanism since Ethernet addresses are nearly trivial to spoof.
-FIXME need examples of config file options for ACLs.
 As you put these values into these configuration files, imagine that you are actually plugging different hard drives into different computers. It's not about moving data to a different drive anymore; it's about moving the drive to where the user needs it to be, and doing so in a completely virtual way.
 ===Boot===
-And that brings us neatly to maybe the most useful thing about a SAN: It makes local storage unnecessary. iPXE supports ATA over Ethernet natively; the DHCP server just has to provide options FIXME
+And that brings us neatly to maybe the most useful thing about a SAN: It makes local storage unnecessary. iPXE supports ATA over Ethernet directly. The DHCP has to provide a suitable root-path option. For isc-dhcp-server, telling a host to boot from shelf 12, slot 9 looks like this:
+  option root-path "aoe:e12.9";
+The DHCP server must not also provide a TFTP next-server and filename. If it does, iPXE will boot via TFTP instead.
+FIXME As far as I can tell, there's no way to have your root volume on ATA over Ethernet. iPXE can use AoE to fetch a bootloader, but that's it: neither Linux nor Windows can use it as a root volume.