Support updating application services in the field with no Internet

Hi,

Can anyone advise if there is a viable solution to application updates once devices have been shipped to customers in the field at locations where no Internet access is permitted?

The problem description:

  • Most customers will have an Internet connection but 5% may not have one available or may dictate that none should be available for security reasons.
  • Balena Cloud has clear advantages and we don’t want to avoid using Balena because of that small number of isolated devices. Not serving that small subset might mean that you can’t be the defacto solution in your market. Or you might have existing loyal customers that have used your product for years and do not want to lose them when rolling out a new product generation based on Balena.
  • So while the standard product enjoyed by 95% of customers can benefit from cloud updates, it is important to still support the customers with no Internet connection on the proviso that (clearly) the cloud features are not available. However, those customers still need to be able to upgrade the firmware (via SneakerNet: USB, SD, local network connection etc).
  • Put simply: for some projects if you can’t support the 5% then Balena is off the table.

Possible solutions:

  • Importing prebuilt images off USB/SD/local network using balena-engine load. This looks promising because it looks like it can be done without shipping devices running development builds. But can balena-engine load only update pre-existing services in the docker-compose file? In only a couple of months, many development projects are likely to have a different arrangement in docker-compose, whether configuration changes, or new services that implement new features etc. If balena-engine load doesn’t support adding new services then how are we to update those field devices based on a docker-compose file that seemed sensible a couple of months ago?
  • Using local mode and balena push could work but it requires you to ship development builds in production devices which has security implications. An upgrade should be possible without an Internet connection: it might not Internet-connected but still sitting on a network somewhere where the network team dictate that there shouldn’t be open SSH or Docker etc ports.
    • Local mode upgrades might be feasible if it was supported in some form without development builds.
    • Builds on the device: I haven’t yet investigated what that exposes you to. A balena deploy command would help here but still means you’re shipping development builds.
    • You could have two software builds (cloud-capable vs internal-only) but from a manufacturing perspective do you really want separate SKUs (development builds for devices that might need local update and safer release builds for the others)? You probably also don’t want to ship all devices with development builds because now you really are opening yourself up to security problems.
  • Instead of trying to do a clever patch / partial upgrade process, just flash a new complete image (preloaded with your application services). i.e. backup the data, upgrade the entire system, then restore the original data.
    • But I haven’t yet seen any clear information about how one would do this.
    • A simple roll-your-own solution risks being unsafe in terms of bad powerdowns. Having A/B copies is likely possible but again I haven’t seen any clear information on how to do this.
  • Install an on-premises OpenBalena server on these sites, but can we expect customers to install a server (capable of running OpenBalena) just to sit alongside / support a single embedded device? It feels a bit clunky this.

Hopefully, that’s a reasonably clear real-world use case that describes the decision process that (at the moment) leads one to the conclusion that as soon as you want to support one customer with no Internet, you currently can’t choose Balena as your platform if you need field upgrades.

What is the best solution? Or is Balena as it stands not feasible if even one customer has no Internet but requires an upgrade path?

I don’t mind putting together my own scripts etc to do something, but I can’t see an easy way to implement any of the possible solutions yet:

  • Local mode and docker push could be used to do a complete multicontainer application update including any changes in docker-compose.yml, addition of new services etc. But they require shipping a device in development mode. If we could temporarily switch a proper release build into local mode and then switch it back into cloud mode then we could ship only release builds and all would be well.
  • The docker save and balena-engine load method of transplanting images doesn’t appear to support changes in docker-compose.yml (adding new services for example). I’m not entirely sure on that yet but it feels like this cannot be used to push changes that one of the developers made in docker-compose, only updates to the pre-existing images built by the service’s dockerfile. A way to change the service composition when off the Internet would solve this, and we developers can find our own way to get the saved images onto the device and call balena-engine load ourselves. At the moment it doesn’t look like there is a local API that can support this.
  • Flash a complete new (preloaded) image while maintaining data: this does sound feasible but is there any documentation about how this could be achieved in terms of partitions that need to be maintained etc? Some hints about how to go about this safely from the perspective of a Balena expert would really help.

It would be great to hear any thoughts on what can reasonably be expected to work.

Many thanks

3 Likes

Hi,

The case of updating the services no devices with no Internet is definitely valid and important. We have it in our roadmap and consider several possible designs to implement it.

Meanwhile, I think the way of re-flashing the device and restoring the app data is the most viable one. But it would have to be implemented on the app side fully.
A potential solution I see is to have a dedicated backup/restore service that mounts the volumes used by other services. Before the update, you could plug a usb drive to a device, trigger the backup procedure (calling some service endpoint) and let it take care of copying the files necessery to store the app state. Then, after re-flashing, the same service could restore the files from the usb stick.

Let us know if this makes sense.