Is it possible to keep two apps / containers on the device for fast switching of the apps?

Product support balenaOS
1

Hey Balena community,

is there a way to enable local caching of balena apps to speedup switching between apps?

Environment & What is the current behavior?

I have two application, one is for quality assurance & sensor calibration, the other one is the normal production mode. These apps are switched a couple of times during the live time of one of our devices. Since each Jetson image is very large ~5GB (cuda, …) switching takes a lot of time. Furthermore after each switch, it appears that the container of previous apps are deleted.

Question

  1. Is it possible to preload an image with two apps, even though only one is selected?
  2. Can i force balena OS to keep container of other apps locally?
1 Like
5

Hello @Langhalsdino thank you for your question!

I don’t think this is possible. You can’t share images between applications because of the authentication the registry and the API provides. Volumes are always purged when moving between applications. Since long time ago we have been talking internally about this, but it isn’t quite there yet.

Do you think you can activate the sensor calibration & quality assurance services with Device Variables?

6

Mhm, it’s a bit difficult, since the containers are not identically, but we probably need to merge them.

Since I build my own Jetson BalenaOS with yocto, could you point me to the part where its deleting the volumes. Maybe i can take a look at it and create a quick hack to make it work for me - Don’t worry, I will share it with the community, too :slight_smile:

2 Likes
7

Ok! let me check with the team what’s the best solution :slight_smile:

1 Like
8

@Langhalsdino could you please explain a bit of detail of your use case? Thanks :slight_smile:

9

As said, it would be great to get more information about your use case, so we can improve the process for developers on the same situation. Having said that, after speaking with some colleagues, the recommendation is to use the release pinning. So you will have a single app and you will be able to pin to a testing release or pin to a production release the devices (all in one or individually) as you want.

Let me know if that works for you!

11

Hello @Langhalsdino how is it going? we spoke last week and I wanted to follow up with you about this. Did you succeed?

I would like to understand as well, if there is any reason why you don’t consider to have 1 unique application where you can have QA + callibration and production services together and moving between them with an environment variable.

If it’s possible help me to understand the blocker that you have to have 2 different applications for this.

Thank you and let me know how we can help you more.

13

Please excuse the long delay.

Sadly i did not proceed on the issue. Since my previous explanation might be a bit hard i try to rephrase it again with another example.

Let’s suppose you have two balena applications for wildlife monitoring. One that observes bees and another one that observe birds. These are fundamental different applications and each quite large due to a lot of dependencies (CUDA, Tensorflow, openCV, …) each with different versions.

Since the wildlife monitoring hardware devices can be used to observe bees and birds, switching the apps occurs often in remote location. While switching between the apps the new container image gets downloaded and the old one gets deleted. Due to the remote location switching of these apps might take days (5-10GB images).

It would be awesome if we could preload the device with multiple apps (e.g. bee-app and bird-app), such that when switching the apps, the other one does not need to be downloaded.

1 Like
14

Would the better approach be to have a multi-container application, where you can choose which container is running at any given time? Given you are already considering storing both applications on the device separately, it seems that your device has the space to include both within a single application.

2 Likes
15

Yes, this is probably the best workaround, our current attempt is using something like:

version: '2'
services:
  bees:
    build: ./bees
    command: bash -c "if [[ $BEES_ACTIVATED == \"true\" ]]; then ./start.sh; else while true; do echo hello; sleep 2; done; fi"
  birds:
    build: ./birds
    command: bash -c "if [[ $BIRDS_ACTIVATED == \"true\" ]]; then ./start.sh; else while true; do echo hello; sleep 2; done; fi"

While BIRDS_ACTIVATED and BEE_ACTIVATED are both balena environment variables :slight_smile:

Even though i am not really happy with this approach, since the other container still allocates some memory and our RAM is a big bottleneck :see_no_evil: .

1 Like
16

Thank you @MParris to jump here with great ideas!

@Langhalsdino now the app changed a bit as the 2nd app is not a QA or sensor calibration anymore but a production app. I think that you can take the benefit of the multi-container apps but it depends on your memory requirements. However I still think it’s the best choice to avoid downloading big chunks of data everytime you move from one to the other.

On the other hand, related with your variables, maybe you can have a variable such as:

$DETECTOR_ACTIVATED === "birds" | "bees" | "bears" | "javelinas"

And another idea is that you can have all this conditions on the start.sh file maybe?

1 Like
19

I somehow missed this thread earlier… @chrisys do you have any other ideas for Frederic here?

This workaround is interesting, but truly having multiple applications on device is the real solution.

21

@dtischler I guess my only suggestion further to what @Langhalsdino already suggested is to have the container exit rather than just idling, that may help to free up the RAM. The containers would need to be set to restart never or perhaps on-failure in the compose file, like this: restart: on-failure. Changing the environment variables should cause both containers to restart in that case, so there’s an opportunity for the correct one to be running after a change.

As you say in the future we’ll be able to have multiple applications, but even in that scenario I’m guessing that both shouldn’t be running at the same time, so the same sort of problem would still exist in that one app would need a conditional start based on some config.

1 Like
22

@chrisys The restart: on-failure policy is probably what i need to free up the ram. This is probably the way to go :slight_smile:

1 Like