I am embarking on a mad science quest to stretch a k3s cluster across a Balena Fleet. Currently my blocker is that when k3s starts, there is a problem due to Balena’s Engine being configured w/ systemd as the cgroup driver.
E1119 23:19:56.803487 23 server.go:290] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"cgroupfs\" is different from docker cgroup driver: \"systemd\"
Is there a reason why cgroupfs does not work with Balena OS that I should be aware of? I found this thread suggesting historical issues, but it’s 2 years old.
Hey @jasonanderson,
Sounds interesting what you are doing! Would it be possible for you the share a repo or a script/snippet that I could use to replicate the issue that you see?
Thanks and regards,
Pranav
Could you share a bit more on how you’re trying to do this?
If you bring your own container engine in a privileged container (e.g. via docker-in-docker DIND, or a similar solution for containerD), then you should be able change cgroup configuration
@pranavpeshwe, @Hades32, sure – there is a repo here and I’ve been finishing a blog post describing this journey, will share when it is complete.
Essentially, if you bring k8s to the mix, while it is indeed possible to run it in privileged mode, there will be issues if the systemd driver is not used to manage cgroups. The reason is that systemd is already managing cgroups for the “undercloud” (balenaEngine), and then k8s would be managing cgroups via the cgroupfs driver. These drivers register cgroup entries in different ways and expect to find information about capacity/usage according to their convention. So if you have multiple containers running on balenaEngine, and then several on k8s, each engine won’t have the right view of the total utilization of the host device. This probably isn’t a huge problem in practice, but in theory it can lead to starvation and over-subscription of device resources. So, I was wondering if it was possible to make balenaEngine use the cgroupfs driver.
The systemd cgroup driver is recommended by k8s docs if k8s itself is running as a systemd service or on a box w/ systemd. This isn’t really the case inside a container unless you install systemd into the container. That is what I have done in my repo, and while it’s a little heavier and more complex than feels necessary, it gets the job done w/o mucking with assumptions on the host device/balenaEngine.
Cheers!
That’s a pretty cool setup that you made, glad that it worked out! Thanks for sharing!
PS: Found you even blogged about that: K3S on Balena
Superb work there @jasonanderson ! Very cool 
Hey, just letting everybody know that I posted some revisions to my blog post, which I actually hadn’t realized was public already 
- it has some more details, in particular my understanding of cgroups evolved over time. It turns out that you don’t need to do the systemd dance inside the K3s container at all, at least I don’t think so with cgroups v2. I’m still working to understand exactly how that’s possible, I suspect it’s something to do with how the unified hierarchy works and perhaps how the /sys/fs/cgroup mount inside the Docker container works. Cheers!