This is an old revision of the document!
Table of Contents
Names
Hosts
A hostname must be pronounced and spelled one way, unambiguously. It should be a member of a set, but not any set another host belongs to. It must be lowercase. It should be possible to type from primitive typing devices like TV remotes.
Set | Hostname |
---|---|
Chemical elements | carbon |
Greek gods | apollo |
Days of the week | sunday |
Medications | pepto |
Animals | duckling, shark (oops; there shouldn't be two in the same category) |
Food | cherry |
IQNs
We do iSCSI Qualified Names by the book, but as simply as possible. A couple of examples:
iqn.2014-08.rocks.narf.sadness iqn.2004-12.com.naptastic.quirk
IQNs can also have :more.stuff on the end. I can't come up with a use case, since hostnames are already fully-qualified and unique.
There are still some :01 suffixes lurking around the network. Fix them or don't; just don't make new ones.
Ideally, /etc/iscsi/initiatorname.iscsi gets the right value as part of system provisioning, and the target only ever has to use the host's correct name. There's a strong warning about changing IninitatorName in that file; if you're changing it to the correct value (i.e., the value in the target's ACL) then it's fine.
NQNs
NQNs: nqn.2014-08.rocks.narf.hostname is the whole thing. No :01 or any of that jazz, at least not yet.
Interfaces
All hosts currently have 1 or 2 Ethernet ports on the mainboard, 0 or 1 wireless devices, 0 or 1 quad-port Ethernet cards, 0 or 1 Infiniband/VPI cards. They all get used differently, so… I'm going to name them differently.
- ethX: Nothing. Linux gives interfaces these names, and I rename them to something else.
- emoX: physical ports on motherboards. (I kinda regret this.)
- wlanX: wireless NICs
- enX: quad-port cards
- ibX: IP over Infiniband interfaces
- mlxX: mlx ports when they are configured for Ethernet operation
Numbers
MAC addresses
I'm not actually forcing MAC addresses on anything… yet. Just making a note that
- NAP is 6e:61:70 in hexadecimal
- nap is 4e:41:50 in hexadecimal
As of September 2024, neither prefix has been assigned to a vendor.
MAC addresses for the quad-port gigabit cards
This was in shark, and is now in a drawer:
hardware ethernet 00:10:18:f8:02:20; hardware ethernet 00:10:18:f8:02:22; hardware ethernet 00:10:18:f8:02:24; hardware ethernet 00:10:18:f8:02:26;
duckling en(0..3)
hardware ethernet 00:10:18:f8:2e:10; hardware ethernet 00:10:18:f8:2e:12; hardware ethernet 00:10:18:f8:2e:14; hardware ethernet 00:10:18:f8:2e:16;
IP Addresses
We use 172.20.0.0/16. The second octet is always 20, until I expand to multiple sites. The third octet is the function number. The last octet is the host number.
There's an almost perfect correlation between interface name, type, number, and function. There's no need for hosts to have extra addressing for extra interfaces. Subnets will be /24 except napwork, which is /22 to accommodate other functions.
- VLANs will be 3000 + the third octet rendered in decimal. (I use nothing outside of 3000..3255)
- P_Keys will be 0xb000 + the third octet rendered in decimal. (I use nothing outside of 0xb000..0xb255. Partitions such as 0xb0ff are in that range but I won't be using them.)
- (0x3000 isn't a valid range for IB subnets since it would result in partial membership. Setting the high bit to 1 gives 0xb000.)
Networks
- 0-3/22: “green”: all emo* and eth* devices should get bridged to this network.
- 0: .1 is the network controller (router, DHCP, DNS); .2+ is for assigned hosts
- 1: .0-255 contains IPMI interfaces and nothing else right now
- 2: .0-255 is for unassigned DHCP hosts
- 3: .0-254 is for VMs that are expected to only run on a specific physical host. Hosts
- .32-39: butler
- .72-79: carbon
- .80-87: duckling
- .88-95: happy
- .96-103: shark
- .104-111: southpark (cudapark, )
- Unused: 4-7 (/22) (if security cameras ever get a network, it should come from this range.)
- Future Multipath: 8-15 (/21 in /24's)
- Unused: 16-31 (/20)
- Unused: 32-63 (/19)
- RDMA networks (/19): IP over InfiniBand and Ethernet can't be bridged, so they need to be separate subnets. I could bridge ib0+ib1 or mlx0+mlx1 but that takes away flexibility so I'm not going to.
- 64: ib0
- 65: ib1
- 66: mlx0
- 67: mlx1
- 68-79 (/22): Reserved
- 80-95 (/21): Reserved
- Unused: 96-127 (/18)
- Multipath: 128-135 (/20 in /24's) (all gigabit right now)
- 128 - en0
- 129 - en1
- 130 - en2
- 131 - en3
- Unused: 136-143 (/20)
- Unused: 144-159 (/19)
- Unused: 160-191 (/18)
- Unused: 192-255 (/18) (remember 255 is valid)
If you add functions please think about whether it makes more sense to assign subnets sequentially, or split a block that's currently unassigned.
There aren't currently any VLANs in use. The Ubiquiti switches might have a stale layout:
- 248: outside
- 256: green
- 384: san0
there might be other san* VLANs defined. I don't remember.
I am no longer able to manage the TP-link switch. It's fixed configuration. I'm not using VLANs so I don't know how it will behave with them.
Host Numbers
- 3: sunday
- 9: carbon (2012)
- 10: duckling (2013)
- 11: happy (2018)
- 12: shark (2020)
- 13: southpark (2023)
- 14: pepto (2026)
- 15..23: (mainframe expanders. If we get bigger than 16 nodes I don't mind renumbering again.)
- 24..31: switches
- 24: butwhy b4:fb:e4:23:52:01
- 25: because b4:fb:e4:89:3f:25
- 26: tplink c0:4a:00:61:39:c0
- 27: (legacy) ngpoe 00:1f:33:fe:8a:be
- 32: quirk
- 33: minecraft
- 34: reclaim [this has to stay put]
- 35: xp-pe (tbh idk if this still exists)
- 36: tf2
Big Idea Dumping Ground
PCIe networking
These products:
- Retimer cards in x4, x8, and x16
- Switch cards in x8 and x16
- Outboard switches with just ports
- Device shelves
- Switch IC
- Stand-up card slots
- OCP slot, maybe?
- PCI and ISA bridges
Better Audio Stack
Want to:
- Connect to all ports in the system
- Dynamically resample to buffers in shared memory
- RDMA to/from other hosts
(is RDMA multicast possible? That would be so fucking sick)
- Full mixing capability
- Backend for jackd
Better DAW
Core concept is a piece of virtual tape that's infinitely long, infinitely wide, and has an infinite number of tape heads that can all be accessed remotely. A recording session is a server you log into, send recorded audio, receive mixed audio, and update a database of what should play when and with what settings.
Better Mixer