Naptastic

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
=====Names=====
====Hosts====
A hostname must be pronounced and spelled one way, unambiguously. It should be a member of a set, but not any set another host belongs to. It must be lowercase. It should be possible to type from primitive typing devices like TV remotes.

^ Set ^ Hostname ^
| Chemical elements | carbon |
| Greek gods | apollo |
| Days of the week | sunday |
| Medications | pepto |
| Animals | duckling, shark (oops; there shouldn't be two in the same category) |
| Food | cherry |

====IQN and NQN====
We do iSCSI and NVMe Qualified Names by the book, but as simply as possible. A couple of examples:

    iqn.2014-08.rocks.narf.southpark:duckling
    nqn.2004-12.com.naptastic.quirk:68

  * 'iqn' for iSCSI or 'nqn' for NVMe-oF
  * YYYY-MM is the month in which the domain was registered.
  * the hostname is in backwards DNS order
  * :initiator or :namespace

From the perspective of a host, every storage bus has a number. Bus 0 is the local PCI bus. Bus 1 is for the host acting as an initiator. Higher numbers are for a host acting as a target. Connections go like this:

  iqn.2004-12.com.naptastic.rocky:01 -> iqn.2014-08.rocks.narf.southpark:68

Since rocky's host number is 68, any other host wishing to share storage with rocky should do so on their own target bus number 68. The network is responsible to connect rocky's bus 01 with southpark's bus 68.

====Interfaces====
All hosts currently have 1 or 2 Ethernet ports on the mainboard, 0 or 1 wireless devices, 0 or 1 quad-port Ethernet cards, 0 or 1 Infiniband/VPI cards. They all get used differently, so... I'm going to name them differently.

  * ethX: Nothing. Linux gives interfaces these names, and I rename them to something else.
  * emoX: physical ports on motherboards. (I kinda regret this.)
  * wlanX: wireless NICs
  * enX: quad-port cards
  * ibX: IP over Infiniband interfaces
  * mlxX: mlx ports when they are configured for Ethernet operation


----
=====Numbers=====

====MAC addresses====
I'm not actually forcing MAC addresses on anything... yet. Just making a note that
  * NAP is 6e:61:70 in hexadecimal
  * nap is 4e:41:50 in hexadecimal

As of September 2024, neither prefix has been assigned to a vendor.

=== MAC addresses for the quad-port gigabit cards ===
southpark en(0..3):
      hardware ethernet 00:10:18:f8:02:20;
      hardware ethernet 00:10:18:f8:02:22;
      hardware ethernet 00:10:18:f8:02:24;
      hardware ethernet 00:10:18:f8:02:26;

in a drawer:
      hardware ethernet 00:10:18:f8:2e:10;
      hardware ethernet 00:10:18:f8:2e:12;
      hardware ethernet 00:10:18:f8:2e:14;
      hardware ethernet 00:10:18:f8:2e:16;

====IPv6====
IPv6 local unique addresses have four parts:
  - 8 bits of fixed prefix (fd)
  - 40 bits of pseudorandom prefix
  - 16 bits of subnet
  - 64 bits of GUID

My top 48 bits are going to be fd20:4e41:5020. Subnet bits should match their Infiniband subnet. Host bits come from the hardware address.

====IP Addresses====
We use 172.20.0.0/16. The second octet is always 20, until I expand to multiple sites. The third octet is the function number. The last octet is the host number.

There's an almost perfect correlation between interface name, type, number, and function. There's no need for hosts to have extra addressing for extra interfaces. Subnets will be /24 except **napwork**, which is /22 to accommodate other functions.

  * VLANs will be 3000 + the third octet rendered in decimal. (I use nothing outside of 3000..3255)
  * P_Keys will be 0xb000 + the third octet rendered in decimal. (I use nothing outside of 0xb000..0xb255. Partitions such as 0xb0ff are in that range but I won't be using them.)
    * (0x3000 isn't a valid range for IB subnets since it would result in partial membership. Setting the high bit to 1 gives 0xb000.)

====Networks====

^ Network ^ IPv4 Range ^ IPv6 Range ^ IB Subnet ^ VLAN ^ VXLAN IP ^ VXLAN ID ^
| green | 172.20.0/22 | fd20:4e41:5020:ffff::/64. | ffff | | 225.172.20.64 | 64 |

  * 0-3/22: "green": all emo* and eth* devices should get bridged to this network.
    * 0: .1 is the network controller (router, DHCP, DNS); .2+ is for assigned hosts
    * 1: .0-255 contains IPMI interfaces and nothing else right now
    * 2: .0-255 is for unassigned DHCP hosts
    * 3: .0-254 is for VMs that are expected to only run on a specific physical host.
      * .72-79: carbon
      * .80-87: duckling
      * .88-95: happy
      * .96-103: shark
      * .104-111: southpark (cudapark, )
  * Unused: 4-7 (/22) (if security cameras ever get a network, it should come from this range.)
  * Future Multipath: 8-15 (/21 in /24's)
  * Unused: 16-31 (/20)
  * Unused: 32-63 (/19)
  * RDMA networks (/19): IP over InfiniBand and Ethernet can't be bridged, so they need to be separate subnets. I //could// bridge ib0+ib1 or mlx0+mlx1 but that takes away flexibility so I'm not going to.
    * 64: ib0
    * 65: ib1
    * 66: mlx0
    * 67: mlx1
    * 68-79 (/22): Reserved
    * 80-95 (/21): Reserved
  * Unused: 96-127 (/18)
  * Multipath: 128-135 (/20 in /24's) (all gigabit right now)
    * 128 - en0
    * 129 - en1
    * 130 - en2
    * 131 - en3
  * Unused: 136-143 (/20)
  * Unused: 144-159 (/19)
  * Unused: 160-191 (/18)
  * Unused: 192-255 (/18) (remember 255 is valid)

**If you add functions** please think about whether it makes more sense to assign subnets sequentially, or split a block that's currently unassigned.

There aren't currently any VLANs in use. The Ubiquiti switches might have a stale layout:
  * 248: outside
  * 256: green
  * 384: san0

there might be other san* VLANs defined. I don't remember.

I am no longer able to manage the TP-link switch. It's fixed configuration. I'm not using VLANs so I don't know how it will behave with them.

====Host Numbers====

  * 3: sunday
  * 9: carbon (2012)
  * 10: duckling (2013)
  * 11: happy (2018)
  * 12: shark (2020)
  * 13: southpark (2023)
  * 14: pepto (2026)
  * 15..23: (future expansion)
  * 24..31: switches
    * 24: butwhy b4:fb:e4:23:52:01
    * 25: because b4:fb:e4:89:3f:25
    * 26: tplink c0:4a:00:61:39:c0
    * 27: (legacy) ngpoe 00:1f:33:fe:8a:be
  * 32: quirk
  * 33: minecraft
  * 34: reclaim [this has to stay put]
  * 35: xp-pe (tbh idk if this still exists)
  * 36: tf2
  * 68: rocky (VM with SR-IOV parts)

----
=====Where Data Lives=====
===Storage===
  * Production volumes:
    * janet: 4TB Samsung QVO.
    * Crucial 120GB SATA SSD boot volume
      * southpark's boot volume
    * Samsung 850 512GB NVMe
      * installed in southpark on the motherboard, I think?
    * Western Digital Black 2TB NVMe
      * installed in southpark on the x16 riser card
      * confusingly mounted at /mnt/sabrent
    * black: Three 4TB Western Digital enterprise drives in a RAID-5 holding backups; currently cold.
  * ZFS pool "truck"
    * Two Sabrent 1TB NVMe drive
      * installed in southpark on the x16 riser card
      * Partitioned partly for I don't remember, but it makes sense
    * four 4TB drives that should really just get recycled
  * Not installed:
    * 512GB Crucial SATA SSD (probably for duckling)
    * 2TB Vulcan Z SATA SSD (probably for shark as a backup)

===So Where Does Shit Live?===
  * VM Images:
    * southpark:/mnt/sabrent/images
      * exported over NFS-RDMA
  * TV shows, movies, software, stuff like that: janet
  * Home directories:
  * OS volumes:

I am still struggling with the fact that things really have to boot off local storage. Fibre Channel without a switch isn't worth it and a switch isn't worth it.

----
=====Big Idea Dumping Ground=====
====SoC on a PCI card====
Mostly this is to enable other projects. It needs at least these features:

  * reasonable amount of CPU and DRAM
  * PCIe 3.0 x1 or better
  * Internal ports:
    - m.2 slot
    - Explicitly copy rpi's 40-pin header
    - LP-DIMM
  * External ports:
    - HDMI
    - gigabit Ethernet
    - (2) USB 3.1

The card presents itself to its host as one or more devices. It can pass through its USB host adapter, network adapter, and m.2 slot, and act as a GPU. It supports SR-IOV. You can boot off it. With the right hardware support, it should be able to stay awake while the host powers off or reboots.

====PCIe networking====
These products:

  - Retimer cards in x4, x8, and x16
  - Switch cards in x8 and x16
  - Outboard switches with just ports
  - Device shelves
    * Switch IC
    * Stand-up card slots
    * OCP slot, maybe?
    * PCI and ISA bridges
  - USB device holder
  - SATA/SAS and NVMe backplanes

The paradigm is "assign devices to hosts" rather than "hosts communicating". The storage shelves don't need to have file services or even know about filesystems. They will show up as a PCI device with storage attached to whatever system they're connected. This is a "peripheral area network" and if CXL becomes part of the mix, it's "memory area network" as well.

Everything except retimer cards needs an API and UI for managing things.

====Better Audio Stack====
Want to:
  - Connect to all ports in the system
  - Dynamically resample to buffers in shared memory
  - RDMA to/from other hosts
    (is RDMA multicast possible? That would be so fucking sick)
  - Full mixing capability
  - Backend for jackd

====Better DAW====
Core concept is a piece of virtual tape that's infinitely long, infinitely wide, and has an infinite number of tape heads that can all be accessed remotely. A recording session is a server you log into, send recorded audio, receive mixed audio, and update a database of what should play when and with what settings.

====RDMA to a Commodore 64====
The 6502 can't drive the C64's memory anywhere near full-speed, but the ROM port has a DMA pin, so a DMA-capable cartridge can do impossibly cool things. I'd like to find a good demo playback engine that already exists, and adapt it to play back data being streamed in over a network.

====Better Mixer====
FIXME