I’ve never kept a devlog before, and I’ll try my best to keep this up as development goes on. I’ve backfilled parts of the journey that I could remember. I got Claude’s help on some of the sequencing. The truth is that I’ve become a little obsessed with this project. It’s crowded out other hobby pastimes for now, but it thoroughly integrates quite a few of my interests at once.

This project will be a long-term pursuit of mine, because it’s currently running my home systems, and thus I must continue pursuing it. It also keeps paying back in utility the more I develop it, which is not true of many of my side projects. It is how I felt about Docket. That has to be a good sign.

Taking the day to establish this devlog and renew a commitment to daily writing feels like a further investment in the project’s long-term health as well.

While there’s still more iteration and polish to do on the media-operator, I can already feel the gravity well of a library browser and management system to rival the venerable Jellyfin. It’s just obvious this is where I’m heading to very soon.

There’s also a wildcard detour I could take with Spotify Soloist to take a swing at music and supplanting Music Assistant. I’m not sure which direction is pulling me more strongly. The wonderful thing about sitting at this vantage point is that I can see so many places to go: whole home media library and playback, a camera-operator to build a K8s-native NVR system, or taking a fresh swing at an autonomous agent built directly into the k3s cluster.

Sometimes, I just don’t know when to quit. I have a pretty solid stack now up through hardware, and yet my Claude subscription hits the weekly reset and I’ve got all these Fable credits to burn. So I start thinking about that whole-home A/V octopus again.

And I really love Kubernetes, by the way, if you couldn’t tell. It’s one of my favorite pieces of software ever. My top four probably go Linux, Kubernetes, Redis, and PostgreSQL. Tried and true, battle-tested systems with clean abstractions and ruthlessly focused on performance and stability. In a lot of ways, liken is my love letter to Linux and Kubernetes.

After the last week or two, there’s an up-and-comer that’s really got my attention: mpv. What a system! Plays literally anything, and you can script the UI with Lua, and you can drive the play over a domain socket. It’s the perfect foundation for a media system following the Linux and Kubernetes ethos.

I kept pressing Claude to cough up ideas for how we could extend this hardware system more broadly. I expressed a whole-home A/V vision, but what I wasn’t seeing at the time was that we were missing a key layer of abstraction between liken and a multimedia fever dream.

liken sees hardware the same way that the kernel does, because, well, that’s all there are, liken and the kernel. The kernel doesn’t see a pair of Bluetooth headphones, it sees a Bluetooth radio that acts as a gateway to an alien network of other devices. It doesn’t see a monitor, it sees a GPU that has two HDMI ports. It doesn’t see audio channels, it sees a sound card with some PCM sinks. I could easily claim the radios and the GPUs and the cards, but they weren’t really shaped correctly to just hand to a workload. A music player doesn’t want a Bluetooth radio, it wants a socket to stream audio to some speakers.

I noticed a common pattern: userspace services that owned a lower-level liken device (like a Bluetooth radio), handled the bluetoothd/bluez user-space side, and then published back to the cluster resources like a game controller or a Bluetooth speaker. This pattern played out the same way across displays, audio, and bluetooth, so I focused on trying to give them as common a shape as we could.

The results are pretty cool! The audio-operator and display-operator didn’t even need any new CRDs because they just claimed a liken resource and republished the child devices as new ResourceSlices in-cluster. Bluetooth was an interesting exploration and I learned a lot about Bluetooth pairing and protocol negotiation.

I think the real joy came when I realized we could have the bluetooth-operator and audio-operator collaborate on exposing things like Bluetooth speakers and headsets. We still don’t have microphones implemented yet (even now in late August) but the groundwork is definitely all there.

Riding high on the success of the homelab migration, I started thinking about what my original vision had been for the homelab, years ago, in the 2010s. I’d pictured a single brain with digital tentacles snaking through the walls and lighting up every corner of the house as part of one expanded digital brain.

I’d realized much of this with Home Assistant, for sure, and Google Home let us pretend we owned a whole home brain and voice. But there was still something very dull about how the homelab had landed as a rack of computers tucked in a corner, with everyone relying on Google TV dongles and speakers rather than talking with our actual house. Now I had a vocabulary to express that whole house vision, liken with follower nodes of any size, exposing hardware devices over DRA for special cases.

The first target for me was our zigbee2mqtt and NUT devices on those Raspberry Pi 2s. With a couple of very low-end mini PCs, they joined as the first non-leader nodes and the Kubernetes DRA system lets me express exactly why the workloads land on the same machines that the UPS and Zigbee radios are attached to.

That entire operation took less than a couple hours, and I realized that there was really no going back.

Due to the sheer humor of the universe, nuc2 (a venerable old beast of 6 years) suffered a power supply failure of some sort, just one week after we rolled the fleet to liken. I ordered a new mini PC to take its place.

The cluster carried on with four machines while the replacement shipped. nuc2’s Machine record read Lost, its scrape targets came out, and the CI runner fleet dropped from five to four.

The replacement, a little Bmax, arrived two days later. Replacing a leader was: boot the stick, read the hardware report, write the new nuc2 manifest, and install it as nuc2. It rejoined the leader rotation the same afternoon. From a dead power supply to a fully replaced leader in two days, and most of that was shipping.

Over the week leading up to the homelab cutover, I spent a good bit of time moving anything that wasn’t bolted down into Kubernetes. It wasn’t that much, just WireGuard (for my homelab’s bootleg Tailscale wannabe system), DNS, NTP, and some metrics things.

I also happened to have a spare mini PC from my days tinkering with an autonomous agent. Yes, I ran openclaw for a week. Yes I bought a dedicated machine for it. Yes I thought I’d build an alternative myself. No I don’t still run any of them today. That mini PC was just begging for a workload, and it became liken-1, the first official physical machine that ever booted to liken. It has still only booted from the USB stick image once, and has only taken liken’s “over-the-air” updates since then.

I resolved that I’d just rebuild the whole fleet on liken.

And I did! By about 11pm that night, the whole thing was done and everything was completely smooth sailing.

And it was…anticlimactic!

LOL nope that’s a total lie. Somewhere in the 7pm hour that evening, when I imaged the first node, it woke up, decided that it owned flux, saw that I had turned the flux addon off on purpose, and proceeded to tear down flux on my entire homelab cluster. This included all the PVs for all the shared files and databases. I panicked and raced to power off that first node but the damage was done. I was heartbroken, assuming I’d be restoring things from backups for hours to come.

Claude, though, Claude said something like, “This is why I took a backup of the cluster’s etcd database before we started”. You did WHAT?

Yep, Claude had quietly taken a snapshot of etcd and put it on my laptop for safe-keeping before we took down the first node. That etcd snapshot had all of the Flux objects and PVCs and PV definitions wiring everything to the right places on our NAS. Claude restored the backup with some cluster surgery, then did a rolling restart and at some point said something like “Okay, now go put the stick in the next machine and hit wipe and reinstall as nuc4. I’ll wait.”

What a champ, and I really wish I still had the transcript from that evening. We really didn’t skip a beat and we really did get done with the full rollout by like 11pm, and I slept like a baby.

After a crazy week of hacking, my conviction really started to sharpen that I had something really interesting on my hands. It was the server Linux distribution I’d always wanted: Kubernetes. We’d made something that I wanted, so it might be something someone else wanted too.

I was still in a mode where I was laser-focused on replacing the distro of my homelab, but I took a beat to make sure the project was real. I made a GitHub org, I gave it a website. I even started the first version of that website on a little 1GB Linode.

Needless to say, I’ve since rethought that deployment architecture. At the time I felt that I should have at least one place to dogfood it for “real”. A cloud provider instance did serve well as a real persistent deployment, but the true endgame was always physical hardware, which would take another couple of weeks.

Heading into a 10-day stretch of relaxation and lake time, I started thinking about catching up on some maintenance on my homelab. My homelab has never been in better shape since the advent of AI coding agents. It has fully automated image updates with flux; the machines themselves were fairly well-maintained with Ansible. It was a shining beacon of Infrastructure-as-Code (perhaps even too much, given some serious circularity in the fact that my internal gitea instance runs on the same cluster it manages, but alas).

But not everything smelled like fresh cut grass here. The five mini PCs backing the k3s cluster were all running Ubuntu 22.04 and had a gajillion pending updates requiring reboots. I was 4 minor versions behind on k3s. My zigbee and UPS controllers were running on Raspberry Pi 2s (yes, 2s) and were literally still running Raspbian, which hasn’t even existed since 2020. I was drowning in updates, and there’s really no good time to take the cluster down to apply them.

I started thinking very hard about why I even had Ubuntu and Raspbian in the mix at all. Just about everything was already running in K8s, except a few oddballs, like WireGuard and bind9 for DNS. Surely any service that can run on a host OS can also run under K8s, right? Why do I even need SSH on these machines? Can’t I just boot straight from Linux to k3s?

It turns out that, yes, yes we can.

Almost.

So the gears start turning: k3s probably doesn’t actually expect itself to be pid 1 and it would be thoroughly unreasonable for Rancher to give it support for that. So maybe we’d just need a tiny little init shim. How does Linux even actually boot up anyway? And how on earth would we give a pod access to a device like a GPU or a zigbee dongle if there’s no udev running? Do there need to be users? What does systemd actually do?

So I started pulling threads to see what I could learn here. And it didn’t take long before Claude and I had a very small proof-of-concept booting up on QEMU. I’m not even sure when I could first run kubectl against it, but from that moment, I knew for sure I was going to take this to its logical conclusion.

(And yes, I did still spend plenty of time with my family by the lake!)