Support updating application services in the field with no Internet

Product support balenaEngine
1

Hi,

Can anyone advise if there is a viable solution to application updates once devices have been shipped to customers in the field at locations where no Internet access is permitted?

The problem description:

  • Most customers will have an Internet connection but 5% may not have one available or may dictate that none should be available for security reasons.
  • Balena Cloud has clear advantages and we don’t want to avoid using Balena because of that small number of isolated devices. Not serving that small subset might mean that you can’t be the defacto solution in your market. Or you might have existing loyal customers that have used your product for years and do not want to lose them when rolling out a new product generation based on Balena.
  • So while the standard product enjoyed by 95% of customers can benefit from cloud updates, it is important to still support the customers with no Internet connection on the proviso that (clearly) the cloud features are not available. However, those customers still need to be able to upgrade the firmware (via SneakerNet: USB, SD, local network connection etc).
  • Put simply: for some projects if you can’t support the 5% then Balena is off the table.

Possible solutions:

  • Importing prebuilt images off USB/SD/local network using balena-engine load. This looks promising because it looks like it can be done without shipping devices running development builds. But can balena-engine load only update pre-existing services in the docker-compose file? In only a couple of months, many development projects are likely to have a different arrangement in docker-compose, whether configuration changes, or new services that implement new features etc. If balena-engine load doesn’t support adding new services then how are we to update those field devices based on a docker-compose file that seemed sensible a couple of months ago?
  • Using local mode and balena push could work but it requires you to ship development builds in production devices which has security implications. An upgrade should be possible without an Internet connection: it might not Internet-connected but still sitting on a network somewhere where the network team dictate that there shouldn’t be open SSH or Docker etc ports.
    • Local mode upgrades might be feasible if it was supported in some form without development builds.
    • Builds on the device: I haven’t yet investigated what that exposes you to. A balena deploy command would help here but still means you’re shipping development builds.
    • You could have two software builds (cloud-capable vs internal-only) but from a manufacturing perspective do you really want separate SKUs (development builds for devices that might need local update and safer release builds for the others)? You probably also don’t want to ship all devices with development builds because now you really are opening yourself up to security problems.
  • Instead of trying to do a clever patch / partial upgrade process, just flash a new complete image (preloaded with your application services). i.e. backup the data, upgrade the entire system, then restore the original data.
    • But I haven’t yet seen any clear information about how one would do this.
    • A simple roll-your-own solution risks being unsafe in terms of bad powerdowns. Having A/B copies is likely possible but again I haven’t seen any clear information on how to do this.
  • Install an on-premises OpenBalena server on these sites, but can we expect customers to install a server (capable of running OpenBalena) just to sit alongside / support a single embedded device? It feels a bit clunky this.

Hopefully, that’s a reasonably clear real-world use case that describes the decision process that (at the moment) leads one to the conclusion that as soon as you want to support one customer with no Internet, you currently can’t choose Balena as your platform if you need field upgrades.

What is the best solution? Or is Balena as it stands not feasible if even one customer has no Internet but requires an upgrade path?

I don’t mind putting together my own scripts etc to do something, but I can’t see an easy way to implement any of the possible solutions yet:

  • Local mode and docker push could be used to do a complete multicontainer application update including any changes in docker-compose.yml, addition of new services etc. But they require shipping a device in development mode. If we could temporarily switch a proper release build into local mode and then switch it back into cloud mode then we could ship only release builds and all would be well.
  • The docker save and balena-engine load method of transplanting images doesn’t appear to support changes in docker-compose.yml (adding new services for example). I’m not entirely sure on that yet but it feels like this cannot be used to push changes that one of the developers made in docker-compose, only updates to the pre-existing images built by the service’s dockerfile. A way to change the service composition when off the Internet would solve this, and we developers can find our own way to get the saved images onto the device and call balena-engine load ourselves. At the moment it doesn’t look like there is a local API that can support this.
  • Flash a complete new (preloaded) image while maintaining data: this does sound feasible but is there any documentation about how this could be achieved in terms of partitions that need to be maintained etc? Some hints about how to go about this safely from the perspective of a Balena expert would really help.

It would be great to hear any thoughts on what can reasonably be expected to work.

Many thanks

5 Likes
13

Hi,

The case of updating the services no devices with no Internet is definitely valid and important. We have it in our roadmap and consider several possible designs to implement it.

Meanwhile, I think the way of re-flashing the device and restoring the app data is the most viable one. But it would have to be implemented on the app side fully.
A potential solution I see is to have a dedicated backup/restore service that mounts the volumes used by other services. Before the update, you could plug a usb drive to a device, trigger the backup procedure (calling some service endpoint) and let it take care of copying the files necessery to store the app state. Then, after re-flashing, the same service could restore the files from the usb stick.

Let us know if this makes sense.

15

Did you ever find a solution?

16

Any updates on the product roadmap concerning this?

17

Hello @rakshak ,

As my colleague Roman mentioned a few months ago, it’s a feature that we want to offer and it’s on our roadmap. However, there are several architectural and security hurdles that we need to solve before this sees the light.
We’ve linked this forum thread with our internal tracking for the offline update feature, so we’ll make sure to let you know once it has been implemented.

By the way, have you tried the side-service approach workaround?

Cheers,
Nico.

20

The side service approach doesn’t squarely address our needs.

In summary, we just need to download the built docker images onto a local file, transfer that file over sneakernet to a device with spotty network, and have that device load in the container image and replace the corresponding old image.
With the understanding that the docker compose itself may change.

I know this is a tough question, but when do you think balena will have something like this out? We don’t need it urgently, but we also don’t want to come back in a couple months and there’s still no clarity on this. Is there anything we can do to prioritize it?

Thank you for your time

21

What you describe is pretty much how updates with Balena already work, it’s just that we go over the Internet. What is “sneakernet”, how it is different from the Internet and what are the advantages of using it?

23

By “sneakernet” I was referring to the process of transferring the file over a USB flashdrive. Where an internet connection is not required from the device side.

I am in correspondence with the Balena team on this so will proceed there. I’ve received confirmation that what I’ve described is currently not supported by Balena but there are plans to do so.

25

@rakshak since you’re in a communication with devs, could you please post a notification in this topic when it’s ready? It’s a handy feature for me as well.

26

I have “sneakernet” upgrades working. Essentially i replicated some of the Balena Supervisor functionality on my own container. The only Balena-managed containers that remained were their own supervisor and a small init container.

27

They posted this guide earlier this year

28

Could you please share your solution? It doesn’t require full device reflashing, right?

29

Yeah, but it talks about data backup, full device upgrade and data restore, whereas you (and me) wanted to reload only specific containers. Did I miss something?

30

That’s right. No reflashing is required. It uses docker save on the build machine to create image files of the container images, and balena load on the target to load the images into Docker.

A set of images are accompanied by a file that contains the image file names, versions etc. I wrote our own version of the Supervisor (I did this using Python) that runs alongside the Balena Supervisor. This is used to do the loading of new images, as well as starting them running, upgrading them etc. Basically replicating the part of the Balena Supervisor that did container start/upgrade handling.

There were two Balena-managed containers: the standard Balena container, and my custom Supervisor. The rest of my application-specific containers were managed by my own supervisor.

I can’t share any code, unfortunately, as this was proprietary work for a client, but I hope that gives you a head-start.

1 Like
31

Travelina’s solution appears like it should work, although I haven’t tried anything similar.
It seems that many would-be docker commands work within the hostOS of the device, where one would swap docker for balena (e.g., balena pull … or balena load …).

In the link I shared in my previous message. There’s a section titled “How to use Offline Updates.” It’s a bit too brief. But a more detailed doc can be found here.
Note that unlike Travelina’s method, this one requires swapping media (e.g., SD card).

32

Yeah, I saw it. I’m not saying it doesn’t work. But reflashing full device memory for every container update no matter how small it is, seems a little overkill to me. Plus you need to take extra step to preserve data.

33

Agreed. Let’s see when Balena officially supports something similar to what Travelina did.