GitLab runner on balena devices for continuous integration testing

Here’s a project I wanted to share with you. I’m very much interested in automation, and recently was playing with the continuous integration/continuous delivery (CI/.CD) tools that GitLab provides. Their platform allows you to add your own “runners”, devices that are running the testing/building/deployment jobs. Inspired by that, I wanted to create an easy way to add balena devices as runners to GitLab (either the platform or your own open source GitLab installation).

The result is the following project:

What it does:

  • the gitlab-runner for a given platform
  • you only have to supply the GitLab runner token to register it, otherwise it is set up with good default values, that you can also change
  • it registers your runner to GitLab, adds device type, architecture labels to your runner (so e.g. you can say "I want this job to be run on a raspberrypi3", or “… on an armv7l device”)
  • the runner is using the onboard docker/balenaEngine, and has an added garbage collector, so that the docker images are not going to fill up your device, hopefully
  • after the first automatic registration you can use gitlab-runner in the container to register the device to more than one projects (or use the GitLab dashboard to assign a runner to multiple projects)

To try it out, you can do the following:

Create a new balena application, and push the above project to it. Then take a device you have, and provision into that application.
If you don’t yet have a GitLab project to test, you can use this following one, just clone it on GitLab:

It has set up a small piece of code (a shell scipt that gets you a piece of advice, in nice colours!), and some testing instructions in .gitlab-ci.yml.
Once you cloned it, go to your projects Settings > CI/CD page, and select Runners. To make sure that you are using this new runner, click Disable Shared Runners on that page. Take note of the runner registration token:

Find the device you provisioned, and add a new service variable for the runner service called GITLAB_TOKEN with the value of your token.

If things go well, your device will automatically register and will show up in a short time, after you refresh the page, something like this:


Afterwards, you can go to to the project’s CI/CD > Pipelines page, where you would see something like this.

(It might happen, that the shared runners already run the job first when you cloned the project before they were disabled, you can clock on a result, and on the next page you can re-run it with the new runner you have).

In the balena dashboard, you might see something like this in the device’s logs, which shows it’s running something:

And this is how the test logs are shown in GitLab:

Congratulations, you should be ready to run build jobs on your own devices!

One of my jobs, run on a Raspberry Pi 3 can be found here:

The way it is set up, the same runner repository linked above works across architectures, I’ve tried on Raspberry Pi 1 (armv6), Pi 3, BeagleBone (armv7), ARTIK710 (aarch64), NUC (x86_64), Edison (i686).

To showcase the multiarchitecture nature of things, I’ve set up a variant of the above demo project, with this little testing rig:

These are 5 physical devices + a Virtual NUC on my laptop, a 5 different architecture, and they are building the test cases according to their device type tags.

You can see the results in this test run:

(feel free to click around, and check the test results, that they were run on the actual device types, shown by the output of /proc/cpuinfo saved with the tests)

Extra Notes

The code sets up the runners by default to use the balenalib/%%BALENA_MACHINE_NAME-debian:latest image (ie. if your .gitlab-ci.yml does not speciy an image, that’s going to be run).

You can see more notes, and the environment/service variables that can modify the behaviour of the automatically registered runner in the readme:

If you want to know more about GitLab runners, check out their documentation and possibly their advanced configuration notes, which might come handy, if you want to register extra runners. That you can do by connecting to the runner service in the web terminal or over balena ssh, and check out gitlab-runner --help.

For further details on the GitLab CI/CD infrastructure, check out their docs.

If you want to check out the runners’ source code, here’s the repository for that too:

What’s next?

There are definitely a lot of things to improve on (configuration, ease of use, etc…). If you find any problem or have an idea for improvement, please open an issue! Or if feeling adventurous, send a Pull Request with your changes! :slight_smile:

Or just ask here of you have any questions, share with us if you have some success story testing code with this project! Cheers!


very nice project! Congrats


Thanks, @gusgonnet, if you happen to use this project for something, would love to hear about it! :slight_smile:

1 Like

@imrehg This is super cool!

I am curious if you had any ideas adapting this to test automatically a balena application on the device?

Would this envolve more of a balena-on-balena approach?

@jalim that’s an interesting thing, my first inclination is that it is not obvious how, but definitely worth a try! We were thinking of doing test with such projects as

in more of a stand-alone mode, but I think it might work in general. Probably not for all the application types (for example not really for those, that require config.txt changes for the Raspberry Pi), and it might require replicating some functionality that is done by the supervisor now in the test script, but might just work…

On the other hand, we are working on a pretty comprehensive on-hardware testing rig + framework, which might be able to bridge the gap. That’s for another day, I guess, this is cool inspiration for what we have now, so thanks! If we make anything useful like that, we’ll shout!

Did a few changes recently, just an update:

32-bit x86 runners

Since the GitLab devs declined to have x86 32-bit helper images, and without helper images the runners don’t work, I’ve done my own little patching on the fork of the runner code, for example here:

and built the relevant images with make helper-docker, retagged things, and pushed them to Docker Hub at The default setup of using these patched images for test devices like Intel Edison, fixed in this commit: Automatic x86 32-bit runner images.

Raspberry Pi 1/Zero balena-engine

It turned out that the binaries we uploaded to the balena-engine releases page for the 17.12.0 release in use do not totally work on armv5/armv6, or at least don’t work the way they should. While debugging that, for those device types I’m installing regular docker for the moment - making the image larger, but also make docker-gc finally work, and the device won’t run out of space anymore, which is good :stuck_out_tongue: Hopefully we can revert this to balena-engine soon.

With this, we have a pretty decent setup, that runs hundreds of tests for our base images, see more details at Automated balenalib base image smoke testing with GitLab runners

1 Like