Kubernetes: Busted again…
It’s actually been a hot minute since my homeprod Kubernetes cluster did anything wayward… and I’ve mostly just left it alone because if it ain’t broken…
Alas, Ubuntu finally pushed out patches for CVE-2026-98432 (a rather nasty bug in TCP/IP that theoretically can lead to RCE with any open socket if the stars align) so I needed to reboot everything. Should be easy enough, drain all the nodes, apply the updates, reboot, uncordon the nodes and wait three to five business days for Mastodon to start again.
Unfortunately I managed to not pin the kubelet package version, and it got upgraded. A smarter individual would have gone “whoops, that was not part of the upgrade plan” and rolled it back, but I am no such individual. I have once again let Kubernetes slip more than one release beyond me, soI cannot upgrade the cluster, but I am pretty good at destroying and rebuilding my cluster these days, so fuck it.
Doubling down on my misfortune was the fact that several things have gone from deprecated to gone in the time I’ve been neglecting it, so it took me nearly two and a half hours to get everything back up and running (and Longhorn still isn’t running, but I haven’t managed to put it into production yet anyway). MetalLB was outdated and refused to start, among other things.
But I’m back up to date now and everything else is running.
In the process though, I learned that one of the fans in my disk server has ceased spinning. It’s a Nidec UltraFlo V80E12BHA5-57 and holy fuck are they expensive. I have them choked the fuck down anyway, so I’m considering my options for switching them out for Noctuas before summer gets here. At the moment I’m just down one fan though.
