Hatching liken

Heading into a 10-day stretch of relaxation and lake time, I started thinking about catching up on some maintenance on my homelab. My homelab has never been in better shape since the advent of AI coding agents. It has fully automated image updates with flux; the machines themselves were fairly well-maintained with Ansible. It was a shining beacon of Infrastructure-as-Code (perhaps even too much, given some serious circularity in the fact that my internal gitea instance runs on the same cluster it manages, but alas).

But not everything smelled like fresh cut grass here. The five mini PCs backing the k3s cluster were all running Ubuntu 22.04 and had a gajillion pending updates requiring reboots. I was 4 minor versions behind on k3s. My zigbee and UPS controllers were running on Raspberry Pi 2s (yes, 2s) and were literally still running Raspbian, which hasn’t even existed since 2020. I was drowning in updates, and there’s really no good time to take the cluster down to apply them.

I started thinking very hard about why I even had Ubuntu and Raspbian in the mix at all. Just about everything was already running in K8s, except a few oddballs, like WireGuard and bind9 for DNS. Surely any service that can run on a host OS can also run under K8s, right? Why do I even need SSH on these machines? Can’t I just boot straight from Linux to k3s?

It turns out that, yes, yes we can.

Almost.

So the gears start turning: k3s probably doesn’t actually expect itself to be pid 1 and it would be thoroughly unreasonable for Rancher to give it support for that. So maybe we’d just need a tiny little init shim. How does Linux even actually boot up anyway? And how on earth would we give a pod access to a device like a GPU or a zigbee dongle if there’s no udev running? Do there need to be users? What does systemd actually do?

So I started pulling threads to see what I could learn here. And it didn’t take long before Claude and I had a very small proof-of-concept booting up on QEMU. I’m not even sure when I could first run kubectl against it, but from that moment, I knew for sure I was going to take this to its logical conclusion.

(And yes, I did still spend plenty of time with my family by the lake!)