Day 12 of #100DaysOfHomelab
June 1, 2023
Another day, another random power outage that caused my UPS to go haywire and not do it's job... In better news I did finally order new batteries for it to prevent the issue from happening a third time.
In the fallout of the power outage, my TrueNAS box wouldn't start it's Kubernetes applications until I switched the source to another disk, restarted and switched it back due to this wonderfully cryptic error
Failed to configure kubernetes cluster for Applications: Missing 'cpuset, cpu, hugetlb' cgroup controller(s) which are required for apps to function
Also my VM for Docker GPU workloads (Plex, Tdarr, Frigate) wouldn't start due to the NVIDIA driver being incompatiable with the kernel. I guess when I had last patched it didnt patch fully so upon booting after the power outage something was fucky.
Thats generally an easy fix by installing the latest driver and installing the nvidia-patch to allow NVENC encoding on consumer graphics cards (in this case a 1050Ti). The harder part is waiting for others using the Plex server to finish their current playback before stop docker to install the new driver.
Monitor your DNS Zones with ZoneWatcher
Be alerted of DNS record changes moments after they happen, not from upset customers.