Remote OS upgrade of custom host image possible?


#1

Hey Guys!

If we create a custom host OS image using Yocto for Rapsberry, is there any possibility to update to OS remotely just like any stock BalenaOS available on the dashboard?

We welcome any solution or idea. Even if we have to run some kind of service on our cloud servers.

Thanks
Peter


#5

Hi there. I will rely on colleagues to fully answer your question, but I wanted to clarify a couple of things:

  • When you say “just like any stock BalenaOS available on the dashboard,” is it a requirement for your custom host OS image to appear as an entry in the dashboard dropdown box that lists the available OS versions to upgrade to?

  • Are you using, or aware of the openBalena project? I am not clear if using openBalena would make it any easier (or harder) to achieve what you intend, but it’s certainly good to be aware of the project. I’ve found this openBalena forum thread that indicates how to perform a host OS upgrade: Roadmap for BalenaOS Updates in OpenBalena
    The hostapp-update script runs balena pull "$HOSTAPP_IMAGE" and I reckon that if it came to it, you could update that script to pull your custom OS image from a different docker repository.


#10

Hello @pdcastro

Thanks for you reply. Basically we want to use your cloud service entirely, but the images we use for flashing the NUC device must be customised. So basically we will build this repo ourselves: https://github.com/balena-os/balena-intel

Flashing our custom image on a NUC device is okay, we can do that, but how can we do remote OS upgrades using your cloud service?

Thanks
Peter


#17

Hey @PeterKAlvin it is not currently set up to have easy updates for custom OS images. It’s possible, but require manual work on your side, and more care if you are doing things differently.

Please also note, that this is not a supported way, and can change in the future without much if any advance notice. We are working on multiple angles on the host OS updates, and thus lot can be different further down the line! Having said that, here’s some usage notes:

Custom OS updates

The main repository is https://github.com/balena-os/resinhup

The updater script run on the device is upgrade-2.x.sh, and the remote update
is done by upgrade-ssh-2.x.sh that invokes the above script on the device, and
supports providing the right parameters for these devices.

It is using regular ssh setup to connect to the device and run the script (we are
investigating, but not done with transitioning this to balena CLI as well).

For the ssh configuration, you likely need to add a setting in your .ssh/config
for the right domain to reach the Balena devices. For balenaCloud, the setting is
this below, where you need to add your username, and the identityfile (for which
you’ve set the ssh public key in the balenaCloud dashboard):

Host balenadevice
  User <USERNAME>
  Hostname ssh.balena-devices.com
  LogLevel ERROR
  StrictHostKeyChecking no
  UserKnownHostsFile /dev/null
  ControlMaster no
  IdentityFile <PATH TO YOUR IDENTITYFILE>

You can test whether your setup is correct by running a connection to a specific
device / UUID with:

ssh -t balenadevice host <UUID>

where you replace <UUID> with the device’s UUID, naturally. This requires the
VPN connectivity enabled on the device, as it goes through the balenaCloud backend.

If that works, the OS update can be done with a command line like this:

./upgrade-ssh-2.x.sh \
   -s balenadevice \
   --hostos-version 2.31.2+rev1 \
   --resinos-repo resin/resinos \
   --resinos-tag 2.31.2_rev1-intel-nuc \
-u <UUID1> \
-u <UUID2> \
...

Here above:

  • -s balenadevice sets which host to connect to, should be the same as you set in .ssh/config in the earlier set (can use any name you like, as long as you use the same here and there)
  • --hostos-version is required, and have to follow this format as above, 2.31.2+rev1 for example
  • -u <UUID> are the list of UUIDs to update, can add as many as you like

Just these parameters are necessary above, and then it would update from resin/resinos Docker Hub repository, and would find the relevant tag by the host OS version and device slug automatically.

The other parameters are possibly required because it’s a “custom” update and depend on your way of doing things.

If you use your own image, have to use:

  • --resinos-repo resin/resinos is the Docker repository to pull the update image from, (more info below), and can modify it to your own. Needs to be a public repository.
  • --resinos-tag 2.31.2_rev1-intel-nuc sets the tag of the image to pull for the update. The tag can be anything in this case, we usually follow a convention of the same tag structure as the regular updates, that is VERSION-SLUG, where in VERSION the + is replaced by _ due to Docker registry limitations. For the NUC the device slug is intel-nuc.

If your device’s slug (the SLUG value in /etc/os-release in the final image) is a custom one, not intel-nuc, then have to customize this.

  • as mentioned above, can use any tag for --resinos-tag, as it is meaningful for you
  • --force-slug intel-nuc internally forces the slug to be the same as the device is currently
    masquerading itself. Required to make proper decisions on e.g. supervisor updates internally. Use the slug the device type that your device’s are using in the dashboard.
  • --assume-supported and --ignore-sanity-checks also relaxes the checks that are done by
    the updater, and by default safeguards regular updates.

You can see more of the options, and maybe more info by running ./upgrade-ssh-2.x.sh --help and ./upgrade-2.x.sh --help for the list of parameters.

To add your own image from a build that you do:

  • You can create your own Docker repository on DockerHub,
    let’s call it my/balenaos here.
  • From the yocto compilation you will get a docker image (not sure what the filename would be, but I think somethign .docker. Load it into docker: docker load -i <FILENAME>, which will give a hash value, something like SHA256:111111...
  • Retag it in docker, using the repository name, the version, and slug you have, for example:
    docker tag <HASH> my/balenaos:X.Y.Z_revW-slug Naturally as mentioned above, you
    can use anything else instead of X.Y.Z_revW-slug, as long as you are clear what it is.
  • Push the image to Docker Hub: docker push my/balenaos:X.Y.Z_revW-slug, and thus
    you are ready to use this image in an update with the script above.

Also have to make sure, that the balena-supervisor version is not modified in your image’s meta-balena layers, as the update will only go to supervisors that are released to our API, and the version set there are released.

This should give all the parts that you need for the update to work, I believe.

The future and some questions

We are working on making it easier to add custom OS images / custom device types to the platform, but it’s a work in progress. When that’s done, our aim is that you can use the regular “self-serve” update method without any custom work like this.

Can you also tell us some more about how your images are customized? What changes you need to do? Just want to check whether that’s something that might be done in the regular device type, or have any relevant runtime customization option that we could add (or have already) in regular balenaOS. Any info is appreciated!


#19

Hello @imrehg

I highly appreciate the details you provided. We are planning a project that needs securing software and data on the IoT devices. Currently if you remove the eMMC or SSD drive from a Balena device, pretty much all data and software can be easily read. We want to employ some kind of disk encryption thats able to use either TPM chip, or CPU’s internal keys and engine for encryption. We love how Balena works, and we already use it at multiple clients, so we are looking for ways to extend the Yocto generated images.

If you have any other suggestion for encryption, please tell us :slight_smile:


#27

The Zymbit solutions are interesting for example, but we have not tried them yet.


#28

The Zymbit solutions are interesting for example, but we have not tried them yet.