Manage/update applications on an offline device

Yeah, the reinstall would likely not be the best way.

  1. the device persistent data cannot really be saved/restored on another device easily. I believe this is on the roadmap as well, but further down
  2. the device ID (and thus the device) you can preserve, by generating a new config.json with the same UUID such as balena config generate --device UUID --generate-device-api-key (see balena config generate --help for more info) and adding that configuration to your image you are using to reinstall the device (e.g. with a preloaded image, so it runs your updated application). Note that the wifi settings of that option are legacy (for balenaOS 2.x need to have separate NetworkManager config files added, tose options by the cli are not effective, we are trying to make this clearer)

For further investigation, though:

  • you can add an ssh key to the config.json https://github.com/balena-os/meta-balena#sshkeys so you can have access to the device even if production device, with your own key locally
  • probably manually / with some scripting, could get the persistent data out of the device
  • if I recall, there was some work being done preloading data out of the device, will have to confirm whether that’s working and if does, how exactly?

With these pieces at hand, with an on the spot preloaded image generation might work as you describe. Will be checking it with our team and get back to you with more details.

Thanks for this input.

  1. Persistent data. Not so worried about this. Yes, I believe I can backup/restore this manually through some scripting.
  2. Device ID: generating a new config does not really sound like a scalable solution. Then I would have to provide a machine specific image to be able to upgrade offline. I would much rather have that the installation could just inherit existing device ID (and preferably inherit the persistent data as well).

But I am looking forward to hear whether your team has other input.

Hi @imrehg

I have tried this balena config generate which outputs the config.json file. Where must this be located?

I have tried to overwrite the /flash-boot/config.json in the base os image, but it seems to be the wrong config.

I am using:

balena config generate --device ${BalenaDeviceUUID} --network ethernet --generate-device-api-key --version 2.39.0 --appUpdatePollInterval 10 --output config.json

and

balena preload ./{IMAGE} --app {BalenaApplicationId} --pin-device-to-release --commit {BalenaCommitHash} --splash-image {SPLASH_SCREEN} --api-token “${token}”

The offline installation create a new device and do not reuse the old device in the balena cloud when connected again

Hi,
Depending on what you want to achieve you could use the CLI to put the config.json in the right place for you.
If you have the os image on the filesystem use: balena os configure <image> or if you have the image already flashed to disk you can use balena config inject <file>,
Kind regards,
Theodor

Tried the balena os configure command which allowed for a old balena cloud device to be reused without creating a new device

in this article https://www.balena.io/blog/remotely-managing-and-deploying-to-iot-devices-at-scale-with-balena/ we have (emphasis added):

In many industry projects, remote access to edge devices is not possible as they are deployed in isolated production networks, without internet access. Because of this, instant updates or immediate help for customers is not generally available. Using balena, the speed at which new features and updates are provided to Bosch customers helps to maximise product usability and reduce time frames between product upgrades and fixes.

It’s possible that the article writer/editor confused a number of features about the networks and the devices. But I was really hoping that this meant that devices on isolated networks could be updated across the network. We have devices sealed inside of IP67 enclosures in production dairy farm milking parlors. Opening up those enclosures to swap SD cards or connect USB keys is not an easy option. Yet weekly/monthly visits to the farms is performed for various maintenance tasks where they could connect to the network and provide updates to balenaOS devices.

Not much to add to the conversation, but just wanted to share our use case in case Balena devs want more details.

Our product will be sold to research companies that may have sensitive information or company policies that prevent a device from accessing the internet. (We know this for a fact from our first product)

I would like a way for the customer to be able to update the application software on their machine. Our product will have a HDMI and USB output. Interested in updating our containers, updating the OS is a nice to have.

My preferred approach:

  1. We instruct the user to download a file from our website in an internet enabled computer.
  2. Customer copies file to a USB and inserts USB into the product.
  3. Customer installs / extracts the file, and the containers are updated on next reboot.

Alternative approach

  1. We send a fully updated flashed SD Card to the customer.
  2. User opens up the unit and swaps out the SD card.

Shipping a SD Card across the world is cheap. This solution is doable, but I dislike it as we would lose persistent information and the user will have to wait days / weeks to do this.

Also dislike it as the user now has to physically open the unit, which has warranty concerns. Implementation details have to be ironed out…i.e - How do we associate the new SD card to the old one? Do they get a new serial number?

Clunky Approach

  1. User opens up the unit and removes the SD Card
  2. User inserts the SD Card into an off the shelf Pi
  3. Connects Pi to the internet and performs an update.
  4. User inserts SD card back into the product

I dislike this approach as it shatters the user experience. (“Oh, they’re just using a Pi”) and now they need physical access to a Pi.

Also, the off the shelf Pi now needs to connect to the internet, which again… may not be possible.


If we can solve for the worst case scenario (their device never connects to the internet or a local LAN) then we have a solution for everything I think.

1 Like

Nice description of the not so nice options. We have exactly the same needs. And I still see the option to “backup device data + reinstall + restore data” as the least poor solution.

@imrehg: Did you check if there is some work to benefit from in regards to what you wrote:

if I recall, there was some work being done preloading data out of the device, will have to confirm whether that’s working and if does, how exactly?

Hi @jason10 @chuyzoz and @krix . Offline updates is something we have been looking at a bit in the recent months. In quoted passage from @jason10, I believe the article was talking rather about balena Machine which is our on premise product that is capable of running completely disconnected from the internet and can deliver updates to lan connected devices all over a private network. This is one solution for the problem described above and allows a user to visit the network periodically and roll out new releases, however this is an expensive option as it requires us managing a onPrem instance for you.

We have been looking very seriously at how to design USB updates, the major factor at the moment is security (from what I understand) as you no longer have the protection of the HTTPS in the transport, so doing a simple implementation of this would open up the devices to be easily taken over. We want to provide a great and secure experience to do this but for now have no ETA as to when this would be available, but from my side I would like us to build it out this year.

Thanks @shaunmulligan. It would be sufficient for our needs to be able to update using a laptop on the local network rather than a USB key. An asymmetric key pair would probably be sufficient.

1 Like

Thank you for that suggestion. We will try to keep that in mind when we make a move at this. Unfortunately there is not yet any active development on this.

I came across

hostapp-update -f

and

update-resin-supervisor -f

in the github PR here: https://github.com/balena-os/meta-balena/pull/1827

Does anyone understand if this describes a solution to this general offline-upgrade problem? I don’t understand the workflow and limitations for this approach.

Hi, those are system level operations that are not supported for production devices. For context, they are used internally by balenaCloud when performing hostOS updates. They could be called locally to perform an unsupported update, but at the moment they still require a network connection. There are a couple of open PRs to support offline hostOS updates (https://github.com/balena-os/meta-balena/pull/1826 and https://github.com/balena-os/meta-balena/pull/1827), but as I say the only currently supported update model is using the dashboard.

Hi, just wanted to know if you are working actively on offline device updates?
Or are there any news about tricks for us to make a reinstallation of a device, but preserve the persistant data incl device ID?

I’d also like to voice my support for this issue. chuyzoz’ preferred approach seems good to me as well.

Some thoughts on security: A public key on the device that confirms the signature of the update is probably enough.

The offline update functionality could be disabled in the supervisor be default and needs to be enabled through config or API that can be accessed only within the docker network or by privileged containers.
This way, we could limit our users to activate offline update capability through an interface that’s only available on the physical device itself and not through the local network.

If you choose to offer this option, I think it’s fair to lay the burden of keeping the device safe on the developer. If physical access to a device by the customer is possible, there’s only so much that can reasonably achieved in regards of security, without involving actual HSMs.