Device doesn't update after installing a factory build

Product support
1

We are running into issues updating our new devices. The installation works correctly but contains an older version (in which one of the services constantly crash and reboot due to a hardware change). A newer version fixes this but we don’t seem to be able to update the devices. We get the following errors:

Failed to download image 'registry2.balena-cloud.com/v2/7a1c542e3da943c5143441dea8f7c790@sha256:ad2f7a063e71b4e9ba3881751502f3d76c84dae0fa94e2e8121fd76348edc546' due to 'Got 500 when requesting v3 delta from delta 
server.'

All our new devices are no longer able to be pinned to any release rendering them unusable.

Some other possibly related erros I have seen are:

  • Failed to install service due to ‘Cannot read property ‘appId’ of undefined’
  • We also previously had a lot of errors where it would say ‘Delta still processing’ and that would usually take hours to resolve.

We are running balenaOS 2.83.18+rev1 with supervisor
12.10.3

2

This could be due to corrupted /var/lib/docker on the device, among other things. Some things to check would be to tail balenad logs (journalctl -u balena -f) as well as supervisor logs (balena logs -f balena_supervisor). These may offer hint as to what’s going on. The nuclear option is to delete /var/lib/docker and restart balena-engine, which would cause the device the re-download/re-install all images based on the target state. A less invasive option is to move the device to an empty fleet (no releases) and then move it back to the original fleet.

4

Thanks, we will try this. We’ve been working arround this problem for a while now. We were also having problems with devices being stuck on ‘processing Delta image’ for a long time. Could deleting /var/lib/docker also solve this by forcing a complete redownload of all the images?