Downloading update never finishes [Resolved]

Yesterday afternoon we pushed an update to our app. Most of the devices downloaded the update quickly, however one of them is having issues. It has been stuck on the updating step for almost 24 hours. It will download part of the image, then restart part way through. Looking at the logs, I see the following entries repeating every 1-30 minutes:
28.11.18 10:47:52 (-0500) Service is already running ‘main sha256:cf22d03cf9a898f168209d74be6d27d0c7d67e85c8b6b4fb40533b6179882f8d’
28.11.18 10:47:54 (-0500) Downloading image ‘registry2.balena-cloud.com/v2/9995b4b712e0a603f6bd2e1f842a6ccc@sha256:61286a5a579d56c7218902f918cf0615e11bfe935c8f4bf77c3334ad4dfd9b09

Has anyone else experienced this before?

Hey Bryan,

That’s interesting. From what you describe, it seems that the device keeps downloading the app version that it currently has, thinking its newer.

Can you confirm that while the device is updating, its already using the latest version? What happens if you push a new update?

While the device is update it is not using the new version. It is running the existing version of the app. I pushed a new version with a whitespace only change.

Now I get the following log entries, and it is still running the software version from the day before yesterday:
28.11.18 12:46:52 (-0500) Failed to download image ‘registry2.balena-cloud.com/v2/53800b1114edf30b34fa4e8943b2d793@sha256:d4c5d126835ce6fd180c5a86f0670fc16a85258269e9dbadf01bfef878254844’ due to '(HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/: net/http: TLS handshake timeout ’
28.11.18 12:46:54 (-0500) Downloading image ‘registry2.balena-cloud.com/v2/53800b1114edf30b34fa4e8943b2d793@sha256:d4c5d126835ce6fd180c5a86f0670fc16a85258269e9dbadf01bfef878254844
28.11.18 12:47:06 (-0500) Failed to download image ‘registry2.balena-cloud.com/v2/53800b1114edf30b34fa4e8943b2d793@sha256:d4c5d126835ce6fd180c5a86f0670fc16a85258269e9dbadf01bfef878254844’ due to '(HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/: net/http: TLS handshake timeout ’
28.11.18 12:47:07 (-0500) Downloading image ‘registry2.balena-cloud.com/v2/53800b1114edf30b34fa4e8943b2d793@sha256:d4c5d126835ce6fd180c5a86f0670fc16a85258269e9dbadf01bfef878254844
28.11.18 12:47:29 (-0500) Failed to download image ‘registry2.balena-cloud.com/v2/53800b1114edf30b34fa4e8943b2d793@sha256:d4c5d126835ce6fd180c5a86f0670fc16a85258269e9dbadf01bfef878254844’ due to '(HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/v2/53800b1114edf30b34fa4e8943b2d793/manifests/sha256:d4c5d126835ce6fd180c5a86f0670fc16a85258269e9dbadf01bfef878254844: Get https://api.balena-cloud.com/auth/v1/token?account=d_1bcab3682d7287612db7b3f30510ff80&scope=repository%3Av2%2F53800b1114edf30b34fa4e8943b2d793%3Apull&service=registry2.balena-cloud.com: net/http: TLS handshake timeout ’
28.11.18 12:47:30 (-0500) Downloading image ‘registry2.balena-cloud.com/v2/53800b1114edf30b34fa4e8943b2d793@sha256:d4c5d126835ce6fd180c5a86f0670fc16a85258269e9dbadf01bfef878254844
28.11.18 12:48:10 (-0500) Failed to download image ‘registry2.balena-cloud.com/v2/53800b1114edf30b34fa4e8943b2d793@sha256:d4c5d126835ce6fd180c5a86f0670fc16a85258269e9dbadf01bfef878254844’ due to '(HTTP code 500) server error - Get https://registry2.balena-cloud.com/v2/v2/53800b1114edf30b34fa4e8943b2d793/manifests/sha256:d4c5d126835ce6fd180c5a86f0670fc16a85258269e9dbadf01bfef878254844: net/http: TLS handshake timeout ’
28.11.18 12:48:12 (-0500) Downloading image ‘registry2.balena-cloud.com/v2/53800b1114edf30b34fa4e8943b2d793@sha256:d4c5d126835ce6fd180c5a86f0670fc16a85258269e9dbadf01bfef878254844
28.11.18 12:54:36 (-0500) Service is already running ‘main sha256:cf22d03cf9a898f168209d74be6d27d0c7d67e85c8b6b4fb40533b6179882f8d’
28.11.18 12:54:38 (-0500) Downloading image ‘registry2.balena-cloud.com/v2/53800b1114edf30b34fa4e8943b2d793@sha256:d4c5d126835ce6fd180c5a86f0670fc16a85258269e9dbadf01bfef878254844

This was resolved by upgrading to balenaOS 2.27.0+rev1

We checked the status of our registry hosts around this time, and nothing was revealed on that side of things. Either way I’m inclined to believe that this was a network hiccup and that you’re upgrading of the device was orthogonal to the image eventually being downloaded.

That being said if you see this again, we’d really like to take a look at the device as it’s occurring, to ensure that was indeed the case.

@CameronDiver We are having the same issue I believe. Is this a hiccup again or should we wait for v2.27? Any update on when this is released?

Hey @Guuz,

v2.27 is already for some devices. Which device are you currently using?

All the best

We are using balenaOS 2.26.0+rev1. I see now that v2.29 is released so i will try that next week!

@Guuz, great!
Keep us posted!

Cheers

It works! :slight_smile: