I understand within an application you can set a device to track the latest release for rolling updates.
I also understand that you can pin a device to a specific release.
Can I pin a device to track the latest release that has a specific tag?
I’ve seen advocation of using tags on devices to update them to be pinned against a specific release.
The staged-releases repo demonstrates use of this api here, and is very helpful:
I’ve also seen on these forums endorsement of using tags on devices and corresponding tags on releases to tie these together:
However, this is not an automatic update like tracking the latest release. I was wondering if features have moved on since, or if this was on the roadmap? (although I admit I didn’t spot it):
Ultimately, we will likely have different release groups such as ‘Alpha’, ‘Beta’ concepts for cohorts of devices and would want to promote releases through testing phases. We also might want to have development releases that are not automatically pushed to the production devices running a particular application. These are all ultimately release versions of the same application.
Without this feature, and without events/webhooks notifying of a new release, I would need to engineer something to poll the Balena API and constantly try to co-ordinate updates of device cohorts to keep them tracking the correct tagged release.
Just want to check if this is the best way right now to achieve this goal, as it seems to have a lot in common with features already available?
What you shared are the latest features related to release pinning. We are not as far as I’m aware working on functionality that would allow you to have different release groups however that doesn’t mean that we would not consider it.
Regarding development releases, we are currently implementing proper release versioning into balenaCloud which as a byproduct will include the concept of “draft” releases. Draft releases will not be tracked for latest so won’t get pushed into any devices set to track latest.
Going back to the multiple release groups, is there any reason creating multiple applications would not suffice for your use case? I can raise this for internal product discussion but I want to better understand the use case first.
Thanks for coming back to me (and sorry for the delay - was off yesterday!).
The fundamental desire is for releases that can be promoted through different groups of devices in a pipeline. Releases staged in a sense of not just applying to the full group of devices automatically, but applying to different groups of devices automatically based on levels of confidence in the release. Think of a release being tagged as ALPHA → BETA → PROD, with devices configured to track different phases, and promotion between these phases being a manual decision.
Ideally, the actual releasing or deployment would not be done by hand (i.e. handpicking test devices or updating static pins). We would just tag a release as the BETA release and it would deploy to the BETA devices.
Applications for Device Cohorts
Creating multiple applications would meet our requirements to some degree. Is this the approach most in line with Balena’s vision, as we’d like to build a fitting approach?
I was under the impression that using differing applications when they actually represent the same application and codebase(s) was not in line with what Balena advocated. This interpretation is based on comments such as:
and I received similar guidance here:
I see two drawbacks with using applications to manage different device cohorts for the same application:
An exploding amount of applications in the Balena dashboard. This is further exacerbated by existing issues we have in managing diversity, and that currently some of our ML models are baked into containers (discussed here).
It doesn’t seem a good fit for flexibly shifting the composition of device cohorts. Adding and removing tags is not very onerous, but moving devices between applications is a more fundamental switch which involves loss of volume data, service env variables, etc. What makes a good set of devices to BETA test a particular release might shift, or may differ for particular releases. Being able to switch tags on the Device is easy and great to denote this - currently it just can’t impact release tracking.
So application separation could work, but I’m not sure it’s the best fit and am reluctant to explode out the number of applications we have in Balena cloud, especially given that we already have quite a lot of applications due to limitations in our ML containers (baked in customer models). Is this the advised route?
Draft Releases
It’s great to see innovations in this area. I actually feel like “draft” releases is a step into this direction, but perhaps not as far as we were desiring.
They would enable us to have a “Production” cohort of devices that track the latest release automatically and are not impacted by releasing builds at lower phases of trust (development/alpha/beta type releases). The one level of separation provides this capability.
They do not go so far as to enable us to have any other cohorts that can track anything else. i.e. ALPHA devices tracking ALPHA releases, or BETA devices tracking BETA releases (draft or not).
With the ability to tag on both device and release sides, and even the staged releases examples from balena using scripting to tie tagged devices to particular releases (perhaps releases of a corresponding tag) - it does feel like the natural relationship and way to implement this. Just wondering if it’s something that fits better on the Balena cloud platform than me trying to tie the two together via some ugly API polling
Please let me know if any other information can help understanding here, and any guidance or push in a certain direction is appreciated.
Hi @louisburton. Looking at your response, I feel like you make some solid points. I think that right now the multiple application approach is a limited workaround (with some obvious drawbacks). I am not aware that we actively recommend that approach, but it seems that you are in a difficult position of having to pick the lesser of two evils. Either a somewhat clunky script or else grouping by application. To me, without knowing the details of your use case, it feels like the script approach is better because it doesn’t (as you observed) involved losing device data. I agree that in any case, it’s not ideal.
I’d like to raise this as something to consider for our roadmap, or at least record the suggestion so that we can track how often it is suggested. Maybe you can give me a bit more insight into your workflow.
I didn’t follow entirely why the polling was necessary. Let’s say an engineer creates a release… at that point, presumably you could run the script that does the tag based pinning, right (presumably for the lowest confidence level). Then as you go through your testing and approval workflow, at some point when tests pass you could run it again to pin the next logical group. This seems a bit hacky, but would it work in theory?
Let me know what you think. Once I really grok your workflow I’ll see if I can present some ideas internally.
We came to a similar conclusion that co-ordinating this via the APIs was the better option. We intend to maintain tags against the Devices representing their release cohort, and tags against the Release representing their release association (their maturity in our testing process).
The actual pinning would then look like finding the release with the given tag, and then pinning the devices to it with a process very similar to update-test-group.sh.
You are correct that polling is not necessary, as long as we are aware when either a release tag changes, or a device release cohort tag changes. We do intend to co-ordinate these actions via our own application layer (using the Balena Cloud API) in the long term, enabling us to audit this also. In the short term however, we will probably do at least part of this via the UI directly in Balena Cloud, and without event hooks we would either need to trust the person doing this manual action to also kick off this synchronisation flow, or periodically poll and sync to prevent any drift from what the corresponding tags indicate a device should be pinned to, and what they are actually pinned to.
Terminology wise, with the recent shift to ‘fleets’, a cohort of devices within a ‘fleet’ seems a natural fit to me, and being able to pin this grouping to a release a natural use.
A subsequent enhancement to support pinning to some form of tag, rather than a forever fixed version number/commit hash, would then allow the behaviour we are desiring.
Hey @louisburton just a ping that I’m digesting this and will see if maybe I can bring it to an internal product discussion next week. Just as an FYI, we are currently releasing the feature for version handling and “draft” releases (this week I think we should see it, parts are going to prod today in fact), although it’s still not really complete yet. With the latest version, you will be able to define a semver release in a balena.yml file in the repo. If you have devices tracking latest, then a “normal” release will become the default. However a “draft” release (created with the --draft option on balena push) will not default for devices tracking latest. So we’re moving in the right directly.
This week is a “no meetings” week for us, so next week will be the earliest opportunity that I will have to present some of these ideas. Feel free to reach out as necessary.
BTW, version 12.46.0 should now support semver versioning and the --darft option. To use the semver versioning, you’ll have to define a file in the root of your source called balena.yml. That file provides meta-information about the project and is what we use for “Deploy With Balena” projects. You can find info about it here: Deploy with balena button - Balena Documentation
Hi, I know it has been quite a while since there has been discussion in this thread, and unfortunately I am not coming to you with any updates about this feature request, but I wanted to let you know that if you are still interested in this feature you should upvote the suggestion for it on our roadmap and share your usecase for it in a comment: Enable staged rollouts of releases across devices · Balena Roadmap
This helps us learn more about what features are highly requested and the usecase our customers have so that we can consider them in internal discussions
