Update when user interacts


#1

Hi all,

For our new product, we use ElectronJS to build an application in combination with the UP Squared. This application is a kind-of POS system. These devices are going to stores all around the world and will be used by people all day. So, updating whenever there is an update is not what we want and have in mind, because then the customer can’t use his/her system anymore when the containers are restarting. And when the ElectronJS container is restarting, the whole application quits and starts again. Again, not what we want.

In earlier products, we didn’t use containers and made a update system, that downloads a package, opens the package and installs what needs to be installed. Using containers is much more flexible and more reliable. But in BalenaOS, whenever I push an update, every device installs that update. I know about the lock files and Balena Supervisor API, but I was wondering if anyone has any suggestions to tackle two problems,

So, TL;DR, 2 questions:

  • What’s the best way to create a function so that my application is in charge when all containers can update? (So when a user clicks on a button and/or when it’s between 00:00 and 03:00 for example)
  • Is it possible to push an update that only certain devices can download until the update is “published”? So like a fleet of beta-devices and when the update is tested properly, it can be published to all devices.

Thanks in advance!


#4

I am aware of three potential solutions:

1- Staged releases
Use/adapt the “staged releases” scripts found in this repo: https://github.com/balena-io-projects/staged-releases

  • “Rolling release” is the feature by which all devices update ASAP when you push an update to your app. The disable-rolling-release-on-fleet.sh disables that.
  • To trigger an update to a specific device when that device’s user “clicks on a button”, I guess the set-device-to-a-release.sh script could be used/adapted. To be honest I’m not sure if the device itself can run that code, but if not then I suppose the device could contact a server of yours, and the server could run the script.
  • Is it possible to push an update that only certain devices can download until the update is “published”?
    Here enters the update-test-group.sh script.

2- Update strategies
An alternative approach that may be suitable in some cases is to change the supervisor’s “update strategy” as described in this page: https://www.balena.io/docs/learn/deploy/release-strategy/update-strategies/

The strategies are “download-then-kill”, “kill-then-download”, “delete-then-download” and “hand-over”. The “hand-over” strategy is the one potentially useful here, as it could deliver zero downtime if the device has sufficient resources and the app’s logic can be modified to accommodate it.

3- Application update locks (“lock files”)
You’ve mentioned you are already aware of this solution, but I wanted to add it to this list for the benefit of anyone else reading this post. Your app can create a lockfile that prevents the supervisor from killing and restarting the app, but updates will still be downloaded to be applied once the lockfile is removed. https://www.balena.io/docs/learn/deploy/release-strategy/update-locking/

Do you think your use case is covered by one or a combination of these options?

Regards,
Paulo


#5

Hi @pdcastro,

Thanks for the quick and detailed response. I will look into all the potential solutions you’ve mentioned!
The staged releases looks like something I will use for the groups, so I can create testing/beta/production groups.

About the update locks, if I understand correctly, the containers will be downloaded but only installed when the update locks are removed? Because that would be a possibility in combination with the staged releases. Downloading the update and only installing it when something is triggered sounds like something that would work and be fast. The only question then is, is it possible to check if there are new containers “waiting to be installed”? Because then I can give the user feedback that new containers will be installed.

Thanks in advance!


#10

@vedicium, a colleague has pointed out that the update locks are created under /tmp and are automatically removed on reboot, so consider that the app might end up being updated if the end user reboots or power cycles it. This may be a good or a bad thing for your use case.
Your understanding matches the documentation, that the containers will be downloaded but only installed when the update locks are removed. I think you’d need to know how long it takes for the the update to be triggered after the lock is removed – it might be OK to wait 5 seconds, but maybe not 10 minutes. I’ll have to carry out some investigation and come back to you on this point, and also on whether / how your app could tell that updates are waiting to be installed. (If it’s not currently possible, it sounds like a useful feature request to be implemented.)

Regards,
Paulo


#11

Hi @pdcastro,

Thanks for your explanation. When the system reboots, it’s currently no issue that the newest updates are going to be installed. However, we would like to have this process in our control. So when we decide in the future, we only want it when the user interacts, we can implement this. Because when the device is booting, and no kiosk is starting, the user thinks something is wrong. So that may be an issue.

Regarding the time how long it takes that an update is installed / triggered to install, that would be a great feature. Or something like a force to install now. Because I can show to the user in the Electron app what the device is currently doing, and it would be useful to show when the device is really installing the updates. And before that whether an update is available, locally (when it’s downloaded already) and remote (when it’s not downloaded but can be downloaded).

Thanks in advance!


#14

@vedicium Supervisor maintainer here, hopefully I can answer your questions.

The update locks themselves haven’t really aged well, and we plan to overhaul them fairly soon, although we’re still in the architecture phase of that.

For your above questions, the length of time until a release to be installed after a lock has been released is not usually determinable, but you can force this by calling the /v1/update endpoint on the supervisor API. This tells the supervisor to trigger the update process, pretty much instantly.

As for detecting when the update is ready to apply, again using the supervisor API, you can check for a downloaded status on the services. The endpoint to call is

curl --header "Content-type:application/json" "$RESIN_SUPERVISOR_ADDRESS/v2/applications/state?apikey=$RESIN_SUPERVISOR_API_KEY"

which will return an object similar to:

{
  "supervisortest": {
    "appId": 1011165,
    "commit": "42a5d01723cac00538fb10df5ea0a671",
    "services": {
      "main": {
        "status": "Downloaded",
        "releaseId": 688486,
        "downloadProgress": null
      }
    }
  }
}

Let me know if this helps you!


#18

@CameronDiver, thanks for the explanation! I’ll let you know if everything works like expected and if I’m missing any features.

Regarding the update locks, you’re not planning to remove them completely? Because my plan was to always have the update lock set, so the containers don’t update unless a user / the custom system asks the OS to. I think this is the correct way to reach my goal?

And since I’m already asking some questions about the updating, what’s the best way to restart another container when a container is updated? The ElectronJS app browses to the NodeJS app (going to http://nginx_container/, and nginx serves the NodeJS/ExpressJS server via a proxy), but when the NodeJS container is updated, the ElectronJS container has to restart of some sort.

I have 2 options in mind:

  • Restart the whole ElectronJS container when the NodeJS container is restarted (I don’t know what the best way is to do this yet)
  • Refresh the ElectronJS browser when it detects that the NodeJS container is offline (Checking connection of socket or have a healthcheck ping to check the NodeJS container version, or both)

And today I had a download loop of an image. I changed the image from node:8-alpine to node:8-slim, because I have a library that requires glibc and doesn’t work in alpine (which is a real bummer because of image size). Only after rebooting the device, the image successfully downloaded. But ofcourse this is not the way the update process must behave. It didn’t say it failed to download, but just restarted the download at 0% at started again up until 50%. I’ve tried it with delta updates on and off, but both failed and restarted the download at around 50%. Do I have to create a new topic for this? Or is this topic just fine?

Anyway, thanks for the response and thinking along for a solution. It makes using Balena for this and future projects a lot better!


#19

Certainly not - we are thinking about how to make them more resilient and helpful, whilst maintaining full backwards compatibility - quite the undertaking :slight_smile:

I would always recommend using staged releases instead of update locks for long lived locks, but different situations call for different methods - so if this works for you then I’d say go for it.

For your other question, we do have update strategies which aim to cover cases like this, you can find out more here: https://www.balena.io/docs/learn/deploy/release-strategy/update-strategies/

For your last point, if the device is not in this state anymore, I couldn’t really say what had happened. If the device gets into this state, you could make another topic and share the logs with me and I might be able to see what’s going on.


#21

We have a very similar need, but haven’t found an easy way of reaching our goal.
We would like a notification to pop up on each device: “software update available”, and it is up to the end user to accept when (or if) he wants to upgrade. Much like your smartphone works.

Our best approach so far, seems to be to setup a dedicated server “upgrade-manager”, that works as a middle man between devices and the balena backend.

A device may then periodically contact the upgrade-manager and say:

Hey I am device “XYZ”, are there any updates available for me?
The upgrade-manager will have all information from the balena-cloud.
The upgrade-manager can have some rules that based on device tags and release tags, will determine which releases should be offered for the device “XYZ”.
When the user accepts an upgrade, the upgrade-manager takes care of requesting the upgrade from the balena-backend.

In principle this functionality can live on each device, but this would require that each device has the high-priviledged access to the balena-cloud. Which might not be a risk you want to take.

We have not implemented this yet, but plan to do, unless we discover a better way.


#22

@krix the method that you describe would be the one that I would recommend - regarding the permissions, you can use the device api key for this purpose (using the io.balena.features.balena-api label, documentation here: https://www.balena.io/docs/reference/supervisor/docker-compose/#labels). You can assign this label to a single service as well, to reduce the reach that other services have.

Balena SDK may also be helpful, as it provides a lot of helper methods for doing this kind of thing.

I assume you’re using pinned releases to do this? The device api key has the permissions to do this.

Your use case sounds interesting, I’d love to hear more about it!