I am having trouble with a container that manages the swap file for our app. I understand that swapfiles are not natively supported by balena, but we require additional memory capacity for running Tensorflow apps on our Jetson Nano-based device, and the internal RAM is falling short with all of the other containers that we also need to run - resulting in frequent crashes of the container / host. This is fixed with the addition of a simple swap container, which I followed the helpful example provided in another similar thread on this forum (https://github.com/muchlearning/make-swap/).
The issue is that whenever I deploy a new release, the host is not able to stop the swap container. The specific error message I get is as follows:
Error removing dead container 'sys-swap sha256:<ID>' due to '(HTTP code 500) server error - container <ID>: driver "overlay2" failed to remove root filesystem: unlinkat /var/lib/docker/overlay2/<ID>/diff/var/lib/swap.img: operation not permitted '
I’m assuming the swap file has some kind of lock on it that is preventing the container from being stopped. Has anyone seen this before / have any ideas on how I can get around it? One thought I had was that if I could run a script on shutdown of the container to execute the swapoff command, but it seems that catching the SIGTERM isn’t the most straightforward approach so would like to avoid if possible.