Running Balena engine inside a container running on balena engine

Product support balenaEngine
1

A question I’m quite sure will initially invoke comments that this is ill advised. Has anyone ever installed balena engine inside of a balena container running on a balena device? The intent being to give users of the device access to their own private balena engine instance without interfering with the base install.

Another option for me may be running Portainer to control the whole device, but use the user accounts in their to limit control of certain components. Has anyone tried portainer with Balena Engine?

3

Note similar steps in Docker but with a custom container: https://hub.docker.com/_/docker/

7

An update here to evolve my question.

I have managed to get Portainer running on the Balena OS and connected to the Balena socket.

Some things look a little off, for example the list of images that are being used by Balena are listed as Unused, presumably because of the way Balena deploys images but can’t be sure.

All in all though it seems to work quite well.

Question then, if I deploy images from Portainer on the device, will Supervisor interfere and try to stop/remove containers, or does it only monitor and act on things started by itself?

8

will Supervisor interfere and try to stop/remove containers, or does it only monitor and act on things started by itself?

I shouldn’t do, no.

Some things look a little off, for example the list of images that are being used by Balena are listed as Unused

Could you explain a little more about this; is this a Portainer related thing? We don’t do anything special when deploying the images, except maybe Deltas but I wouldn’t expect anything to be detectable in the way these work.

10

Thanks for the info.

In terms of it being ‘a little off’, that did indeed end up being a Portainer related issue. I had hidden the containers related to Balena using Portainers labels, and subsequently Portainer hadn’t handled the associated images very well. I logged it with Portainer: https://github.com/portainer/portainer/issues/3565

So far so good, things seem to be running well, and your confirmation that Supervisor won’t intervene in these created containers is reassuring.

Deploying containers through Portainer and adding a Balena label doesn’t seem to take affect, for example the Supervisor doesn’t become available with “io.balena.features.supervisor-api”: “1”. It does though if I redeploy one of the images that are on the device as part of its initial Balena Push, and repurpose them with new labels. This is actually a benefit for me, but wasn’t what I had anticipated. It seems when I do this manually without Portainer it doesn’t take affect either, not sure if that is expected? Is there something injected into the containers built with Balena Push that makes them work with the labels?

Would rather the title of this thread now be Portainer related, but unfortunately can’t seem to change it.

For those searching Portainer, here is the Docker Compose that worked for me:

version: '2.1'

services:
    portainer:
        ports:
            - '9000:9000'
        labels:
            io.balena.features.balena-socket: '1'
        volumes:
            - 'portainer:/data'
        image: portainer/portainer
        command: -H unix:///var/run/balena-engine.sock

volumes:
    portainer:
12

@maggie0002 Hello, I’m checking in and seeing if your Portainer deployment is still working?

I’m very interested in doing the same.

13

It is. Apologies this thread ended up a little convoluted. I didn’t end up running balena engine in balena engine, but gave users access to the device’s own balena engine via Portainer.

Here is it in action, check out docker-compose.yml and the Portainer folder: GitHub - LearnersBlock/learners-block: An open source project that lets individuals and organisations provide their educational resources, websites and apps to users offline.

If it is the balena in balena thing you are after, I’m pretty sure I saw a balena repo put together for this now.

14

Looking at it again now, you may be better off with the example in this thread. In the repo I copy in a pre-configured config file which probably isn’t necessary for most people.

The repo might be useful though to see an example of how I handled getting the right architectures. Balena doesn’t pull from docker hub based on the device it is deploying to, but the device it is executed on. So I had to use dockerfiles.

15

Thanks for your docker-compose.yml above; it worked for me.

My objective was to have a bootstrap OS that contains Portainer. Then on top of that, I can customize any number of containers, start, stop, and update each independently without affecting the other containers. BalenaCloud does not seem to include this feature.

Thanks again for the help!

1 Like
17

Just a note: As the Balena Engine is based on Docker and having it run separate from the host would not allow e.g. /dev/ access or any other Balena features that are depending on labels, I wonder if your use-case wouldn’t better be server by DIND (Docker-in-Docker)?
The DIND images ( Docker Hub ) work just fine on Balena. You only need to mark the container as priviledged

1 Like
20

To clarify, I think you are saying labels and other Balena features won’t work in either scenario, rather than they will in the docker-in-docker scenario so docker in docker is worth considering. It read a bit like labels would work on docker-in-docker.

What would be the benefit of docker-in-docker for the use case? Seems there is a working balena engine to tap into already. Wouldn’t running docker-in-docker on top only consume more resources, prevent image and cache sharing, and result in a performance hit for the containers in docker?

Would be good to hear of any pros, as I did go back and forth on the idea initially.

For my core system containers that I didn’t want people to see I used Portainers ‘hidden’ label to prevent tampering.

21

I did just think of one. I guess Balena Engine on the host won’t support docker-compose files above v2.1? Docker in docker would. That would be nice to have I guess.

22

I’m not clear in what you are suggesting.

The setup I have now is the generic x64 device image, with portainer deployed as an application to that Application/device. It appears that BalenaOS is giving me a docker interface to manipulate with Portainer.

Are you saying that containers that I load through portainer onto the docker instance that BalenaOS is providing won’t have access to /dev/ ?

24

Hello Matthew,
I initially understood that you wanted to run balenaEngine inside balenaEngine. My comment was just about that, i.e. if you want nested containerization (for more isolation/security) using the ready made DinD images would certainly be easier. Also, a nested balenaEngine might not be able to offer what it normally gives you over plain Docker.
If you’re fine with sharing the container engine between your users and your app, there is no reason to use DinD.

26

I’m okay with sharing the container engine between users and my app, so loading Portainer into a device image seems the quickest way to go. Thanks for the clarification.

28

@maggie0002 are you able to ssh into the Portainer container after using the docker-compose.yml?

I get the following errors when trying to access the console from Balena Cloud:

OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory": unknown
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"bash\": executable file not found in $PATH": unknown
29

Oops, this appears to be a design decision with the Portainer docker image:

30

It is a Portainer thing. But it depends how you started it. I think if you start it using a Dockerfile you may be able to sh inside, at least i can on my setup although haven’t investigated fully why.

31

On a not directly related note it would be worth being aware of this issue: Evaluate container's restart policy and exit status before starting · Issue #1668 · balena-os/balena-supervisor · GitHub

In my setup i would stop the Portainer container when it’s not being used and then this issue became relevant.