Enable staged rollouts of releases across devices

Described internally as “Release Channels”, we have a spec in progress for facilitating release channels like dev, beta, testing, production, etc.

Chris Crocker-White: For those upvoting here - please take a few minutes to describe your use case for staged rollouts if you can, it will only serve to help us prioritize and develop the functionality.

I would imagine this like Git branches, where releases are uploaded to a specific branch/stage which would simplify pushes from CI/CD. Each stage does have a separate rollout strategy.

Examples for rollout strategies:

  • full: all devices at once
  • staged: like on Apple App Store (“Release update over 7-day period using phased release”)

Devices are assigned to a stage by default (like main or production), but can be moved afterwards.

This feature would prevent confusing fleet titles like “myfleet_test” and “myfleet_dev”.

Anuj Deshpande: The way to do this currently is to use tags. Each device has a tag and a value - for example release_plan is the tag name and I have setup values like alpha, beta, and production for my devices in this fleet. In the fleet view, where I see all devices, I can create filters to list devices based on this tag - and then pin all these devices to a particular release.

https://docs.balena.io/learn/deploy/release-strategy/release-policy/#pin-device-to-a-release

We would like to define more release policies in addition to the default. Instead of pinning all non-default devices, all devices could follow a release policy that does reference a release.
Currently we have implemented this in our management application, all devices do belong to a “deployment ring”. This results in a lot of device pinning request to the balena API. They could be reduced to a set-release-request per release policy.

Hello,

Basically our workflow is the following:

  • Develop a new feature
  • Deploy it on few devices (usually “beta”)
  • Test it for a few days
  • Deploy to all fleet

I think @Anuj Deshpande way of doing this is very nice. We could maybe just automate this by making devices track release with specific tags for example ?

When I create a release I could flag it as beta (usually automatically though CI, when merging on main branch). Then all device pinned to the “latest with beta tag” would update.

Once I’m ready to update all fleet, I can make an other release, usually from a tag through CI, with a speciufic number like 1.0.0 and a production tag. Then it would update rest of the fleet (tracking latest with production tag).

Vipul Gupta: Thanks, everyone for sharing their use-cases. Please keep them coming. We have been looking at this from the testing side as well in reference to https://roadmap.balena.io/posts/45/release-with-confidence-testing-releases-on-balenacloud

With release pipelines and testing, we intend to make iterations faster and much more automated.

We currently use seperate fleets for production & staging releases.
Both fleets have identical environment variables and configs, except for the DATABASE_URI and NODE_ENV (which we set to production or staging to enable some features in our app)
To deploy to a fleet, we make a pull request from master into either production or staging. Upon merging, Bitbucket pipelines kicks in and pushes to the corresponding fleet.

Looking forward to the Apps feature (https://roadmap.balena.io/posts/7/enable-apps-single-and-multiple-to-be-installed-on-a-fleet-multi-app) to see how this workflow will be affected.

This support thread covers our use case - Devices Cohort Pinned to Automatically Track Release Tag

Ultimately, we want to pin differing releases to differing groups of devices.
We don’t want to overload ‘fleets’, these devices may be part of the same grouping, running the same application. However, we don’t want to release to all devices in a single big bang, we want to be able to release in a staggered fashion to this fleet. An Alpha, Beta, etc. approach to help reduce the risk of releases. Ideally it would be nice to have canary type releases with progressive roll forward and automatic roll back, but any form of release grouping would be nice.

We accomplish this right now with a lot of orchestration on top of Balena that uses the Balena API. Effectively constantly hitting the API with complex queries to identify the devices grouped by tags, and the corresponding release for that tag - sync’ing the two together via individual device release pinning.
This takes knowledge of this completely out of Balena, and it also makes it hard for us to keep things in sync as expected.

Chris Crocker-White set the status to No Status

Just wanted to chime in here and say that the work to enable apps (both single and multiple) detailed here will need to happen prior to this request: https://roadmap.balena.io/posts/7/enable-apps-single-and-multiple-to-be-installed-on-a-fleet-multi-app

The work there is likely to partially solve for some use cases for this feature and will move release management to the app concept instead of the fleet, and so it would seem prudent to tackle that first rather than adding complexity to the release workflow as it stands and then having to move it all around with the addition of apps.

The general high level plan is that fleets should become an organizational unit for devices, and apps an organizational unit for releases. A decent chunk of functionality requested here should fall naturally out of this work, then we can review what else is required here and layer that on top of what we implement for apps.

Also wanted to thank everyone here for upvoting and commenting - it’s great to have all this interaction with you guys; long may it continue! Cheers!