Hi, the data partition is indeed full. I had a quick look at its content and the bulk of it are docker application layers. I would suggest looking at balena images and cleaning up unused images, but the balena-engine is not running and cannot start probably because of the lack of space in the data partition.
My approach would be to remove the docker layers, restart the engine and the supervisor and let the application to be re-downloaded. However, this means removing application data from the device so it might lead to data loss, and also will stop the application from working for a while.
There have been roadmap discussion about monitoring the disk space and let the administrator know once it reaches a threshold, but unfortunately I don’t have an ETA for this.
Let me know how you want to proceed,
there’s no serious loss if the docker layers are removed and the app re-downloads, so you can proceed. just let me know of the things you did please, so if this re-occurs - i can fix it. thanks again.
Hi again, so as neither the balena engine or supervisor were running I went ahead and removed the /var/lib/balena/{overlay2,containers,image,tmp} to make some space in /mnt/data. After that I restarted the balena engine and re-installed the supervisor and the application is downloading now.
As I mentioned before, this was a extreme case as balena-engine had stopped and could not be started. We would usually analyze the docker layer and remove the ones that are not needed any more.
Once the application finishes downloading the device should be functional. Please let us know if there is any further problem.
Hi, I can see this device is in the process of updating. It’s quite slow and I’ve been having some issues connecting to the device over the VPN. What are the network conditions like where this device is, though it looks like it should complete without any intervention reasonably shortly.
This looks network related in that the download can’t complete and is restarting. In addition, as I noted previously I’m having a hard time connecting to the device. Does this device have a stable network connection?
yeah there’s 8 other devices on that network. the other 8 updated balenaOS just fine as well as app deployed today. only these 2 i mentioned above are stuck in this download loop. i can provide access to another machine that’s on the same net - working fine if you want.
Hi, I managed to run the device diagnostics on these devices and they are both experiencing very slow disk writes. This is likely manifesting itself in other issues such as my inability to reliably connect to the device. What SD cards are the devices running, as it is possible they are failing. Do you have the ability to swap out one of the SD cards to test?
yeah we are going to replace the sd cards. meanwhile i did a rm on /mnt/data/docker and issued a reboot from the console, so everything gets recreated. so far it’s downloading fine. will update once it’s done.
