I’ve seen this on several devices recently. We will push a release, and the BalenaCloud console will indicate “downloaded” for all the services, but some of them (always the same group) will not start.
According to the supervisor log when I attempt to start the service the API call gets a 404. And surely enough, in /mnt/data/docker/containers there’s no directories corresponding to the containers that won’t start.
Rebooting the device often makes this problem go away, but I had one recently where I had to push another release, and then reboot and then the containers folder was updated.
I guess my question is, can anyone comment on how my devices are getting into a state where Balena thinks the services are downloaded but they don’t appear to be?
The group of services that have the problem are all quite large in size but the devices are on a high-quality ethernet connection and there’s no obvious (at least to me) networking errors, and no space issues on my /mnt/data partition.
These devices are Generic_x86_64 or Intel_NUC, Balena OS 2.68.1 with supervisor 12.3.0 or 12.3.5, though also one device with 2.4.6 / 10.6.27 (which we have some of in the field)