Device types, Multiple open fleets with single git repository?

What is the best way to support multiple device types with open fleets? We want to support both raspberry pi and jetson devices, and make the fleets openly joinable. It seems like this can be supported using the dockerfile templates and multiple fleets.

However, to make the fleets open, it seems like each need their own git repository and balena.yml file? We’d like to avoid duplicating code across multiple repos like that.

For the moment I’ve worked around it by using contracts instead, but it’s a bit clunky, and I still have to copy+paste a lot of code across services.

Hey @msherman sorry I didn’t see your post earlier!

The answer here is two-fold. Firstly, multiple device types are definitely supported within a single open fleet, with the significant caveat that they need to share the same architecture. I’m assuming you’re thinking about multi-architecture so I’ll go into that in a moment but to start with let’s look at the functionality we have today.

When you create a fleet in the balenaCloud dashboard and choose a default device type, that fleet is set up for the architecture of that device type, and hence when you go to add a device (again, within balenaCloud), the device types that are offered to you are ones that support that architecture. This also includes backward-compatible device types. For example, if you create a Pi Zero based fleet, you’ll be able to add a whole bunch of different devices including the full range of Pis and Jetson devices. This is because they’re all backward-compatible with the Pi Zero. However if you were to create for example a Jetson Nano fleet, you would only be able to add aarch64-compatible devices, of which the Pi Zero and other older devices are not, so you’d have less choice here.

The above functionality translates directly to balenaHub with some additional functionality. If we again take the example that I create a Pi Zero fleet in the dashboard, the available device types will be the same as above (i.e. include all supported devices) but on balenaHub this list will be filtered by the list of ‘supportedDeviceTypes’ that is provided in the balena.yml file pushed with the release. The idea behind this is that you may have only added support for specific device types to your fleet and so do not want users adding other devices where the software may not work correctly.

In addition, it looks like you’ve correctly identified that you can support multiple different architectures by using Dockerfile templates and multiple fleets, but you do not have to have a separate repo per-fleet. You’ll see on balenaHub, especially when it comes to blocks, there are many examples where the architecture variations have been published but they’re all based on the same underlying repo.

So, taking these examples, you have the option of supporting all the device types within a single, backward-compatible-architecture-based fleet or you can create a separate fleet for each architecture and use the same repo and push the exact same code to each fleet.

Aside from all of this, we’re aware that we need to support true multi-architecture fleets, and that’s definitely something that’s in the roadmap for hub. Think of the future feature to be something that essentially decouples supported architectures from a fleet, so you can have one fleet and turn on support for all different architectures. That way when you push your release, the builders will know to build not only for the architecture of the default device type, and on the open-fleet side on hub it will be presented as a fleet that you can join with a much larger list of available device types.

I hope this helps clarify how things are working, and thanks for using balenaHub! Let me know if you have any more questions :slight_smile:

1 Like

Thanks for the info! I think I can provide some more context, and ask more specific questions now.
The repo in question is GitHub - ChameleonCloud/chi-edge-workers: Balena fleet to deploy workers for CHI@Edge

For now, we’re targeting exclusively aarch64, so I don’t think we’re doing “multi-arch” in that sense. However, we are targeting a number of different devices, and need to build and load kernel modules for each device.
For the most part, the rpi3 (on 64bit), rpi4, and pi cm4 run the same kernels, and the same code, but the jetson nano and jetson xaver nx need their own modules loaded, and especially large cuda libraries added to the nvidia devices.

In addition, we wanted these fleet(s) to be freely joinable, as per the “open” fleets.

I initially tried to have a single repo, and used the dockerfile.template format to push to multiple fleets, one per device type. However, it seems that to be open, each fleet needs a git repo associated with it, said repo having a balena.yml containing the fleet name. chi-edge-workers/balena.yml at bff2c6b31f766a0aabab2c72136700c19161b946 · ChameleonCloud/chi-edge-workers · GitHub

Based on that limitation, I combined our “per-device-type” fleets into the unified fleet linked above.
Of course, I could be off base on exactly what is required to make a fleet “open”, but that seemed to be the crucial step.

1 Like