incorrect current release and unable to update

Product support
1

I suddenly have this on two devices

The current release shows ‘factory’ and no services show up in the dashboard. ssh’ing into the host and running balena stats does show my services running and the application seems to function normally.

rebooting does not get the system out of this state

Failed to download image 'registry2.balena-cloud.com/v2/aaa0383c8649766203d15ed6c3887324@sha256:f88eb01638d2c6bbecc29e978617b688c7c7cf9771494519cae0683e19eec2cc' due to 'loading delta base image "sha256:d0b01ad7854dad329240089c2e8894e8813610ebdaf87274ee44af3bbf3fc999": failed to get digest sha256:d0b01ad7854dad329240089c2e8894e8813610ebdaf87274ee44af3bbf3fc999: open /var/lib/docker/image/overlay2/imagedb/content/sha256/d0b01ad7854dad329240089c2e8894e8813610ebdaf87274ee44af3bbf3fc999: no such file or directory'

Please note that application can not be updated anymore. Any help from support here would be ver much appreciated.

2

Hello @bas thanks for reporting the issue and apologizes for the experience.

Could you please confirm the device type, OS and supervisor versions? Do you have any clue why this could happen or what triggered this? (e.g. 2 releases very close in time? a very large release?)

In the meantime, I will check for a temporary solution that we have that can help you to fix the issue!

Thanks

3

@bas find here a temporary solution meanwhile we can try to understand why this is happening.

You can download the target image yourself with the instructions below. If you continue to see this problem, please let us know again. It should not occur, and we are trying to find the root cause.

To download, first you need to get the balenaEngine CLI on the hostOS to authenticate by using the device’s credentials with the commands below. Execute them one by one. You should see confirmation that the login succeeded.

deviceApiKey=$(cat /mnt/boot/config.json | jq -r .deviceApiKey)
uuid=$(cat /mnt/boot/config.json | jq -r .uuid)
endpoint=$(cat /mnt/boot/config.json | jq -r .registryEndpoint)

balena login ${endpoint} -u d_${uuid} -p ${deviceApiKey}

Then you can trigger the pull by copying the registry2.balena-cloud.com/v2/... url that is surfaced in the error and running:

balena pull registry2.balena-cloud.com/v2/...

Keep in mind that this is a complete image pull without any deltas, but like in the case of the above devices you might be able to benefit from image layer caching.

You can find the location of all the image URLs that are part of the target release in the Supervisor State section of the Diagnostics tab on the dashboard.

The device should then be able to apply the update on the next interval that the supervisor triggers an apply, but you might actually prefer to trigger a supervisor restart so that it attempts to apply the update sooner than later while troubleshooting. You can do that with:

systemctl restart balena-supervisor

Let us know if this works!

4

devices affected

  • 03b5abf22d9db704286dc4da1543444b
  • 65d733bc1dfbf62fc25eb3fbee052e6a

i’ve opened up 03b5abf22d9db704286dc4da1543444b for remote support if you want to have a look. that one is currently idle.
both

  • pi4
  • supervisor 14.13.4
  • os balenaOS 4.0.26+rev1

i’m not exactly sure how this could have happened. i’ve been doing some intense development but these particular devices are pinnen to a fixed release.

thanks for that info. I got the images downloaded (no login required by the way) and rebooted balena-supervisor and it got further. after “Delta still processing remotely. Will retry…” it reverted back to the same state (but with different image hashes).
So i pulled another image and rebooten balena-supervisor… After a few services start to be killed and reinstated. Now my list of service is back and current release shows the correct version again…

think I am out of this pickle! Thanks

1 Like