New build available, but devices are not downloading it

Hi,

I’ve noticed our devices stopped checking and downloading updates pushed to git.
Issue began right after the platform overhaul (that introduced multi-container support).
Currently, I have a successful build (according to the “releases” tab) that is marked as current. Still, my test-device is not downloading it (it is currently running the previous successful build).

Thanks!

Hi @ymaia, sorry for the inconvenience, there is indeed an issue we are currently working on deploying a fix.
thanks
Shaun

Yes, I was considering that. You guys must still be working trimming all the edges.
That device did downloaded the updated build a few minutes ago, so things should be getting back to normal. Thanks!

Hey there @ymaia, thanks for your patience. Everything should be back in place as of the last few hours. Let us know if you are still experiencing any strangeness.

Thank you. As of this morning, things look to be working, though I am seeing two things:

  1. On a single application, I’m seeing this longer than I expected:
Failed to download application 'registry2.resin.io/v2/eb5d5940*******27d9bc5' due to 'Got 401 when requesting image from delta server.'
  1. At a high rate, I see builds being “cancelled” unexpectedly at the image upload step: This build has been cancelled

I’ve detected a related issue, in which devices that employ delta-updates download the build all the way to 100%, but the install never happens.
Yesterday, I had a 100% download that installed after a few hours. But today I have a completed download that has not installed spontaneously even after 10 hours.

Hey @ymaia could you give us a link to the device and grant support access to it and we can look into the issue.

Hi Shaun,

I’ve private messaged you with the access details. Support access has been granted for 24h.

Hello, the device is now fixed and has pulled the update successfully. The issue was a regression in Supervisor 6.6.3 that sometimes prevents the delta update from finishing, that’s fixed in 6.6.9. I updated the device to that version and you should not face any more issues. Let us know how it goes.

Good, I’ll test this, but for now the delta install has finished successfully.
I’ll upgrade the base OS as soon one is available, in order to avoid this issue on our other devices.

Thanks!

I have a similar issue I think, I have 4 ‘deployments’ of my application. Whenever I add a device it grabs the latest release availabe but old devices do not pull the new release it seems.

I’m running the Intel NUC images both on “Resin OS 2.12.0+rev1 (dev)” running v7.1.7 with a multicontainer ‘starter’ project.

Is there a way to manually ‘force’ the update?

hello,
what do you see in the dashboard logs? Do the devices even try to pull the update?

1 Like

The older board that has been up for 2 days shows nothing in the logs, it’s missed 3 updates now.

I actually have another application on an RPI3 which hasn’t seen an update either (on release 1 of 3) so this doesn’t seem to be limited to NUCs.

Note that one weird thing is none of my boards will respond to reboot or restart saying "Device not found: " even though I can get to the host terminal and the application runs so I know the connection is good.

hello,
can you send us a link to your dashboard and grant support access in order to investigate more please? (via a private message)

Hi,

Regarding the device that was updated to 6.6.9:
Our development team had another build that was pushed today. This one downloaded to 97% and then became stuck.
We’ve rebooted the device from the resin dashboard. That caused the download to restart and, this time, the update did finished correctly.

@arfoll, I see that the devices received some updates now. Can you check if those are the correct updates now?

So one device got the extra container that wasn’t there but it’s still not been updated, it’s just displaying cb6e170
-> c252174 and running -> downloaded? One of the boards hasn’t been updated in 2 days so do I have a misconfiguration? Should I be doing something manually?

besides the status (“running -> downloaded”) which will be fixed, all your services (exept one) are on the same commit now, which I believe it is the latest. Is that correct?

Well not according to the dashboard? I could do an update to check the actual container has been updated but just this seems odd?

Hi,
The two devices that you granted access to, are now on the latest release (c252174).
We’ve also released resinOS v2.12.3 which fixes the issue you ran into.
Upgrading to the latest version will keep you from seeing this in the future.
You can use self service hostOS upgrades for your prod devices, which is available in the actions page of each device.
Keep in mind that hostOS upgrades aren’t available for the dev variant of resinOS though.

1 Like