Hi, I’ve made a small project to show how to achieve canary deployment / incremental roll-out of Balena releases using GitLab CI/CD. The project can be found here. I’ve included the project’s README as a reference:
We’ll add canary deployment to your existing Balena-based application using a few steps:
- Define the canary stages in your Balena application.
- Add required files to your repository.
- Configure the GitLab CI/CD pipeline.
- Creating a new release that uses canary deployment.
Releasing new features to remote IoT devices carries risks. To mitigate this risk, canary deployment can be used. Instead of updating the entire fleet at once, updates are first released to a small subset of the fleet, say, 10% of the devices. After some time delay the update is released to an increasing number of devices, until all devices have been updated. By following this strategy, there is a higher chance of catching issues during the deployment process. The deployment process is usually performed in an automated fashion using a technique called Continuous Deployment (CD).
Balena is a tool for IoT developers that takes care of (cross-)building your Docker-based application and robustly deploying it to devices in the field. Balena devices are grouped by applications. Out-of-the-box, there is no support for canary deployment. However, by combining the Balena CLI and Python SDK, it is quite easy to extend functionality for this use case.
This demo project shows how to combine GitLab CI/CD and Balena to set up canary deployment for your application. The implementation is easy to port to other CI/CD providers using the provided tips.
Add canary deployment to your existing application
Note: it’s assumed your project is hosted on gitlab.com and your Balena application is available from the root of your project (i.e., you can run
balena pushfrom there).
Define canary stages
You can define the canary stages by adding a tag to each device in your Balena application. To assign a particular device to stage
N of your deployment (
N = 1, 2, …), set the following tag:
In this example, it is assumed that there are 3 stages, although it is easy to modify the code for fewer or more stages. A possible distribution for deployments with 3 stages is:
- 10% of your devices:
- 30% of your devices:
- 60% of your devices:
Finally, ensure the release policy of your application is set to “Pinned to…” instead of “Tracking latest”. This ensures that during your first canary deployment, your devices don’t immediately update (since they haven’t been pinned yet).
Add required files to your application repo
Add the following files from this project to the root of your repo:
.gitlab-ci.yml: defines build and deploy jobs, as well as the delays between them.
deploy.py: script to pin a set of tagged devices to a new release.
Dockerfile.ci: defines Docker image for the CI/CD runner. Contains Balena CLI and Python SDK.
Setting up GitLab CI/CD pipeline
In the GitLab project, go to “Settings” -> “CI/CD” -> “Variables” and add two environment variables:
BALENA_API_KEY: a valid API key for your Balena account.
BALENA_APP_NAME: the name of the Balena application.
Then, go to “Packages & Registries” -> “Container Registry” and note the tag, denoted by
$GITLAB_CONTAINER_TAG here. Example:
registry.gitlab.com/pascal.hwky/balena-canary. Build and upload the Docker image defining the runtime of the CI/CD runner to this tag by using the following commands:
docker build -t $GITLAB_CONTAINER_TAG -f Dockerfile.ci . docker push $GITLAB_CONTAINER_TAG
.gitlab-ci.yml with a text editor and update the values denoted by
TODO. The default values are used to demonstrate canary deployment with the simple application defined in this project (a single
Dockerfile with an infinite sleep command).
Pushing a new release using canary deployment
Instead of manually running
balena push, create a release with canary deployment by simply committing to your Git repository. The GitLab CI/CD pipeline will, by default, run automatically on all branches. To restrict deployments to specific branches (commonly
master), add the
only keyword to
.gitlab-ci.yml (see docs).
Customizing the deployment process
As described in the section below, this project contains a minimal implementation. It is likely that you need to extend the functionality by modifying
Integrating with other CI/CD providers
Although this example runs on GitLab CI/CD, the core of the implementation depends on the Balena CLI and the Python script
deploy.sh. Therefore, it should be trivial to port this project to other CI/CD providers. Some things to consider:
- The commands that should be run are defined for GitLab CI/CD in
.gitlab-ci.yml. Other CI/CD providers will have different formats. Ensure that there is a way to delay execution of deploy stages and pass the release ID between stages.
- The runtime environment needs the Balena CLI for
balena pushand the Balena Python SDK for running
deploy.py. In GitLab CI/CD, this is ensured by using the Docker executor with an image built from
- The Balena API key and application name need to be exposed to the runtime. In GitLab CI/CD, this is handled by adding runner variables.
Since this is a demo project and not intended for production usage, there are a few limitations:
- If a release takes longer to build than the time between the
buildstage and the first deploy stage, the entire deployment will fail. This is because
deploy.pyasserts whether release status is “success”. You should include additional checks and wait loops to allow for longer builds.
- There is no check to see if all devices in the specified application are updated after the last stage. Two situations in which this might not be the case are when a device is added during the deployment, or when devices are not properly tagged with
rollout-stageor the associated stage is higher than the number of stages defined in
- The call to the Balena Python SDK in
deploy.pyare a bit slow. Since each device is processed sequentially, the deployment can take long for stages with a large number of devices. Consider adding concurrency to speed up.