Manage/update applications on an offline device

Wauv, I am happy to see there is a big interest for this. I think this could be a killer feature for Balena. And without being too dramatic, the lack of offline upgrade options could potentially be a showstopper for us, forcing us to migrate to another solution.
Is there any thing we as a community can do to help or encourage work on this?

1 Like

Thank you for the feedback everyone, this feature request is in our radar and I’ve bumped it up for a product discussion. We will keep this thread updated.

Thank you for the feedback everyone, this feature request is in our radar and I’ve bumped it up for a product discussion. We will keep this thread updated.

Is it possible to (mis)use local mode to upgrade a device which is offline? I found that some new features have been introduced around local mode, so I wondered if I am overlooking a possible work around?

Besides that: Any news about this from your roadmap discussions?

1 Like

Its certainly possible to use local mode to upgrade a device that is not connected to our API, provided you can connect to it using a local IP. However local mode requires that you use a development image and for security reasons, development images should not be used in the field ( see https://www.balena.io/docs/reference/OS/overview/2.x/#dev-vs-prod-images ).
I’ve reached out to the rest of the team to see if they have suggestions that I’m not aware of and we’ll update you once we have more information

1 Like

Hi,
looks like offline updates - e.g. from USB - are on the road map but are not expected to be implemented this year.
So for now it seems your best chance would be local mode or getting the devices online for a short period of time.
Only other option I could think of would be using your own ‘trusted’ environment using open-balena: https://www.balena.io/open/. It is probably a bit of an overkill though…
Regards
Thomas

Yeah, I think local mode is a no-go, because of the security concerns.
Another work-around I am considering is to do a reinstall from a USB disk image, just like I normally do when setting up a new device.
The challenges here would be:

  1. How can I preserve or restore the /persistent partition in a safe way?
  2. Can I preserve/restore the device ID and other persistent Balena data as well?

Yeah, the reinstall would likely not be the best way.

  1. the device persistent data cannot really be saved/restored on another device easily. I believe this is on the roadmap as well, but further down
  2. the device ID (and thus the device) you can preserve, by generating a new config.json with the same UUID such as balena config generate --device UUID --generate-device-api-key (see balena config generate --help for more info) and adding that configuration to your image you are using to reinstall the device (e.g. with a preloaded image, so it runs your updated application). Note that the wifi settings of that option are legacy (for balenaOS 2.x need to have separate NetworkManager config files added, tose options by the cli are not effective, we are trying to make this clearer)

For further investigation, though:

  • you can add an ssh key to the config.json https://github.com/balena-os/meta-balena#sshkeys so you can have access to the device even if production device, with your own key locally
  • probably manually / with some scripting, could get the persistent data out of the device
  • if I recall, there was some work being done preloading data out of the device, will have to confirm whether that’s working and if does, how exactly?

With these pieces at hand, with an on the spot preloaded image generation might work as you describe. Will be checking it with our team and get back to you with more details.

Thanks for this input.

  1. Persistent data. Not so worried about this. Yes, I believe I can backup/restore this manually through some scripting.
  2. Device ID: generating a new config does not really sound like a scalable solution. Then I would have to provide a machine specific image to be able to upgrade offline. I would much rather have that the installation could just inherit existing device ID (and preferably inherit the persistent data as well).

But I am looking forward to hear whether your team has other input.

Hi @imrehg

I have tried this balena config generate which outputs the config.json file. Where must this be located?

I have tried to overwrite the /flash-boot/config.json in the base os image, but it seems to be the wrong config.

I am using:

balena config generate --device ${BalenaDeviceUUID} --network ethernet --generate-device-api-key --version 2.39.0 --appUpdatePollInterval 10 --output config.json

and

balena preload ./{IMAGE} --app {BalenaApplicationId} --pin-device-to-release --commit {BalenaCommitHash} --splash-image {SPLASH_SCREEN} --api-token “${token}”

The offline installation create a new device and do not reuse the old device in the balena cloud when connected again

Hi,
Depending on what you want to achieve you could use the CLI to put the config.json in the right place for you.
If you have the os image on the filesystem use: balena os configure <image> or if you have the image already flashed to disk you can use balena config inject <file>,
Kind regards,
Theodor

Tried the balena os configure command which allowed for a old balena cloud device to be reused without creating a new device

in this article https://www.balena.io/blog/remotely-managing-and-deploying-to-iot-devices-at-scale-with-balena/ we have (emphasis added):

In many industry projects, remote access to edge devices is not possible as they are deployed in isolated production networks, without internet access. Because of this, instant updates or immediate help for customers is not generally available. Using balena, the speed at which new features and updates are provided to Bosch customers helps to maximise product usability and reduce time frames between product upgrades and fixes.

It’s possible that the article writer/editor confused a number of features about the networks and the devices. But I was really hoping that this meant that devices on isolated networks could be updated across the network. We have devices sealed inside of IP67 enclosures in production dairy farm milking parlors. Opening up those enclosures to swap SD cards or connect USB keys is not an easy option. Yet weekly/monthly visits to the farms is performed for various maintenance tasks where they could connect to the network and provide updates to balenaOS devices.

Not much to add to the conversation, but just wanted to share our use case in case Balena devs want more details.

Our product will be sold to research companies that may have sensitive information or company policies that prevent a device from accessing the internet. (We know this for a fact from our first product)

I would like a way for the customer to be able to update the application software on their machine. Our product will have a HDMI and USB output. Interested in updating our containers, updating the OS is a nice to have.

My preferred approach:

  1. We instruct the user to download a file from our website in an internet enabled computer.
  2. Customer copies file to a USB and inserts USB into the product.
  3. Customer installs / extracts the file, and the containers are updated on next reboot.

Alternative approach

  1. We send a fully updated flashed SD Card to the customer.
  2. User opens up the unit and swaps out the SD card.

Shipping a SD Card across the world is cheap. This solution is doable, but I dislike it as we would lose persistent information and the user will have to wait days / weeks to do this.

Also dislike it as the user now has to physically open the unit, which has warranty concerns. Implementation details have to be ironed out…i.e - How do we associate the new SD card to the old one? Do they get a new serial number?

Clunky Approach

  1. User opens up the unit and removes the SD Card
  2. User inserts the SD Card into an off the shelf Pi
  3. Connects Pi to the internet and performs an update.
  4. User inserts SD card back into the product

I dislike this approach as it shatters the user experience. (“Oh, they’re just using a Pi”) and now they need physical access to a Pi.

Also, the off the shelf Pi now needs to connect to the internet, which again… may not be possible.


If we can solve for the worst case scenario (their device never connects to the internet or a local LAN) then we have a solution for everything I think.

1 Like

Nice description of the not so nice options. We have exactly the same needs. And I still see the option to “backup device data + reinstall + restore data” as the least poor solution.

@imrehg: Did you check if there is some work to benefit from in regards to what you wrote:

if I recall, there was some work being done preloading data out of the device, will have to confirm whether that’s working and if does, how exactly?

Hi @jason10 @chuyzoz and @krix . Offline updates is something we have been looking at a bit in the recent months. In quoted passage from @jason10, I believe the article was talking rather about balena Machine which is our on premise product that is capable of running completely disconnected from the internet and can deliver updates to lan connected devices all over a private network. This is one solution for the problem described above and allows a user to visit the network periodically and roll out new releases, however this is an expensive option as it requires us managing a onPrem instance for you.

We have been looking very seriously at how to design USB updates, the major factor at the moment is security (from what I understand) as you no longer have the protection of the HTTPS in the transport, so doing a simple implementation of this would open up the devices to be easily taken over. We want to provide a great and secure experience to do this but for now have no ETA as to when this would be available, but from my side I would like us to build it out this year.

Thanks @shaunmulligan. It would be sufficient for our needs to be able to update using a laptop on the local network rather than a USB key. An asymmetric key pair would probably be sufficient.

Thank you for that suggestion. We will try to keep that in mind when we make a move at this. Unfortunately there is not yet any active development on this.

I came across

hostapp-update -f

and

update-resin-supervisor -f

in the github PR here: https://github.com/balena-os/meta-balena/pull/1827

Does anyone understand if this describes a solution to this general offline-upgrade problem? I don’t understand the workflow and limitations for this approach.

Hi, those are system level operations that are not supported for production devices. For context, they are used internally by balenaCloud when performing hostOS updates. They could be called locally to perform an unsupported update, but at the moment they still require a network connection. There are a couple of open PRs to support offline hostOS updates (https://github.com/balena-os/meta-balena/pull/1826 and https://github.com/balena-os/meta-balena/pull/1827), but as I say the only currently supported update model is using the dashboard.