It’s been a while that I’m playing with Docker and Balena engine on RPi to find a good solution for deploying a safe production environment that can protect and survive from SD card corruption.
As far as I understand and experiment, there is no way that I can make the /var/lib/docker{balena-engine} directory read-only or using overlayfs with the lower read-only dir.
On the other hand, when you run a container with docker or balena, with every single process that gets executed in the docker container, I can observe a disk write to the device that mounted at /dev/lib/docker. I also tried to run the container with --read-only option but it did not make any difference.
Since balena-engine is being used in the Balena OS, how can you assure that the SD card corruption on the partition that holds the docker/balena-engine data-root won’t affect the whole solution?
Hi,
yeah you can’t really avoid those disk writes that happen when you start a container, but they would only happen at the beginning and end of your container lifecycle, which in most cases shouldn’t be too often.
A bigger problem are usually the writes done by the application itself, there it’s best to mount the location that these writes happen in the container as tmpfs to keep them in memory.
Thanks for your response. However, I use iostat and monitored the write to the disk while a container is running. What I observed was different. I observed write for every single command I was running in the container. as simple as “ls” command.
I just recorded a video. on top, I have balena-engine-daemon running. in the middle, I’m running a container and at the bottom I have “iotop -o” running. (the environment is RPi)
I’m unable to view that video on our forums, would it be possible to upload the video to youtube, or share the file itself somehow? It’s worth noting that running command such as ls do incur disk I/O, there’s the reading of the initial binary, then the reading of the directory structure, for example.
I am no expert but looking at the video you provided it seems like disk IO happens when you perform the command the first time, which makes sense because the system needs to read the binary from disk, but then subsequent runs of the same command like ls don’t do IO because they are in cache. Could you run that test?
I recorded another video. In this video, I have mounted a separated partition (/dev/mmcblk0p6) on /var/lib/balena-engine to make sure that I only capture the disk I/O from balena-engine processes. The result is still the same. I can see Disk write to the disk when running arbitrary commands in a container.
Hi @zixet, my teammate maintaining balenaEngine (robertgzr) notes that it’s expected (using binaries will incur I/O). If you want to make absolutely sure that the OS will never write to SD card, perhaps balenaOS is not a great fit. Sounds like you need an OS (like Alpine Linux) that can run entirely on memory.
If you are going to use balenaOS, I guess the solution for protecting against SD card corruption would be having regular backups.
Thanks @gelbal ! I wish docker/balena could run on overlayfs
I could have achieved this (zero disk I/O) using Podman on top of overlayfs. Podman could be a potential replacement for Docker in this use case.
