Running application on a docker

pli · January 25, 2020, 12:40am

Hi,

Currently my project is running on raspberry pi devices. I am wondering if I can run the application (balena supervisor and my application containers) as docker containers on an appliance. Do I need balenaOS in order to run the balena supervisor?

Thanks
Patrick

roman-mazur · January 27, 2020, 10:38am

Hi Patrick,
It’s technically possible to run supeervisor without balenaOS. But you would need to figure out a bunch of parameters to start it. Here’s a quick reference to how it’s configured in balenaOS:

github.com

balena-os/meta-balena/blob/development/meta-balena-common/recipes-containers/resin-supervisor/resin-supervisor/start-resin-supervisor#L73


    hasValueChanged "BALENA_ROOT_CA"        "$BALENA_DECODED_ROOT_CA" "$SUPERVISOR_CONTAINER_ENV_JSON" || \
    hasValueChanged "NODE_EXTRA_CA_CERTS"   "$NODE_EXTRA_CA_CERTS" "$SUPERVISOR_CONTAINER_ENV_JSON"; then
        echo "Container config has changed!"
        return 1
    else
        echo "Container config has not changed"
        return 0
    fi
}

runSupervisor() {
    balena rm --force resin_supervisor || true
    balena run --privileged --name resin_supervisor \
        --restart=always \
        --net=host \
        --cidenv=SUPERVISOR_CONTAINER_ID \
        --mount type=bind,source=/var/run/balena-engine.sock,target=/var/run/balena-engine.sock \
        -v "$CONFIG_PATH:/boot/config.json"  \
        -v /mnt/data/apps.json:/boot/apps.json \
        -v /resin-data/resin-supervisor:/data \
        -v /proc/net/fib_trie:/mnt/fib_trie \

I would suggest running balenaOS in a container instead, which should be much easier.

Using the NUC image (e.g. resin/resinos:2.46.0_rev1-intel-nuc) on x86 machines usually works well.

pli · January 28, 2020, 8:20pm

Hi Roman,

Thanks for your reply. I tried the balenaos-in-container but ran into problem. What I did was:

created a new application with device type Intel NUC.
downloaded the config.json file for device type Intel NUC.
on the docker running on windows (for my testing), I created a container using command

.\balenaos-in-container.ps1 -image resin/resinos:2.46.0_rev1.dev-intel-nuc -id test -c “$PWD\config.json” -detach

dockers logs -f shows
[FAILED] Failed to start Balena Application Container Engine.
See ‘systemctl status balena.service’ for details.
[ OK ] Stopped Balena Application Container Engine.
[ OK ] Closed Docker Socket for the API.
Stopping Docker Socket for the API.
Starting Docker Socket for the API.
[ OK ] Listening on Docker Socket for the API.
Starting Balena Application Container Engine…
[FAILED] Failed to start Balena Application Container Engine.
See ‘systemctl status balena.service’ for details.
[ OK ] Stopped Balena Application Container Engine.
[ OK ] Closed Docker Socket for the API.
Stopping Docker Socket for the API.
[FAILED] Failed to listen on Docker Socket for the API.
See ‘systemctl status balena-engine.socket’ for details.
[DEPEND] Dependency failed for Bale… Application Container Engine.
[ TIME ] Timed out waiting for device /dev/ttyS0.
[DEPEND] Dependency failed for Serial Getty on ttyS0.
[ OK ] Reached target Login Prompts.
[ OK ] Reached target Multi-User System.
Starting Update UTMP about System Runlevel Changes…
[ OK ] Started Update UTMP about System Runlevel Changes.

On the balena.io dashboard, it didn’t show any device attached to the application.

Opened a bash shell to the balena os docker and got this error when running “balena image ls”

bash-4.4# balena image ls
Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?

Thanks

garethtdavies · January 28, 2020, 9:08pm

Hi, the first thing to check is that you have fulfilled all the prerequisites https://github.com/balena-os/balenaos-in-container#prerequisites. Assuming that, on the device, I’d take a look at the logs so can you post the output of journalctl --no-pager --unit balena.service to see if it has any clues?

pli · January 29, 2020, 7:52pm

Hi @garethtdavies,

Here are the logs:

level=warning msg=“Running experimental build”
level=warning msg="[!] DON’T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON’T KNOW WHAT YOU’RE DOING [!]"
level=info msg=“libcontainerd: started new balena-engine-containerd process” pid=45119
level=info msg=“starting containerd” revision= version=1.2.2+unknown
level=info msg=“loading plugin “io.containerd.content.v1.content”…” type=io.containerd.content.v1
level=info msg=“loading plugin “io.containerd.snapshotter.v1.aufs”…” type=io.containerd.snapshotter.v1
level=warning msg=“failed to load plugin io.containerd.snapshotter.v1.aufs” error=“modprobe aufs failed: “modprobe: FATAL: Module aufs not found in directory /lib/modules/4.19.76-linuxkit\n”: exit status 1”
level=info msg=“loading plugin “io.containerd.snapshotter.v1.native”…” type=io.containerd.snapshotter.v1
level=info msg=“loading plugin “io.containerd.snapshotter.v1.overlayfs”…” type=io.containerd.snapshotter.v1
level=info msg=“loading plugin “io.containerd.metadata.v1.bolt”…” type=io.containerd.metadata.v1
level=warning msg=“could not use snapshotter aufs in metadata plugin” error=“modprobe aufs failed: “modprobe: FATAL: Module aufs not found in directory /lib/modules/4.19.76-linuxkit\n”: exit status 1”
level=info msg=“metadata content store policy set” policy=shared
level=info msg=“loading plugin “io.containerd.differ.v1.walking”…” type=io.containerd.differ.v1
level=info msg=“loading plugin “io.containerd.gc.v1.scheduler”…” type=io.containerd.gc.v1
level=info msg=“loading plugin “io.containerd.service.v1.containers-service”…” type=io.containerd.service.v1
level=info msg=“loading plugin “io.containerd.service.v1.content-service”…” type=io.containerd.service.v1
level=info msg=“loading plugin “io.containerd.service.v1.diff-service”…” type=io.containerd.service.v1
level=info msg=“loading plugin “io.containerd.service.v1.images-service”…” type=io.containerd.service.v1
level=info msg=“loading plugin “io.containerd.service.v1.leases-service”…” type=io.containerd.service.v1
level=info msg=“loading plugin “io.containerd.service.v1.namespaces-service”…” type=io.containerd.service.v1
level=info msg=“loading plugin “io.containerd.service.v1.snapshots-service”…” type=io.containerd.service.v1
level=info msg=“loading plugin “io.containerd.runtime.v1.linux”…” type=io.containerd.runtime.v1
level=info msg=“loading plugin “io.containerd.runtime.v2.task”…” type=io.containerd.runtime.v2
level=info msg=“loading plugin “io.containerd.monitor.v1.cgroups”…” type=io.containerd.monitor.v1
level=info msg=“loading plugin “io.containerd.service.v1.tasks-service”…” type=io.containerd.service.v1
level=info msg=“loading plugin “io.containerd.internal.v1.restart”…” type=io.containerd.internal.v1
level=info msg=“loading plugin “io.containerd.grpc.v1.containers”…” type=io.containerd.grpc.v1
level=info msg=“loading plugin “io.containerd.grpc.v1.content”…” type=io.containerd.grpc.v1
level=info msg=“loading plugin “io.containerd.grpc.v1.diff”…” type=io.containerd.grpc.v1
level=info msg=“loading plugin “io.containerd.grpc.v1.events”…” type=io.containerd.grpc.v1
level=info msg=“loading plugin “io.containerd.grpc.v1.healthcheck”…” type=io.containerd.grpc.v1
level=info msg=“loading plugin “io.containerd.grpc.v1.images”…” type=io.containerd.grpc.v1
level=info msg=“loading plugin “io.containerd.grpc.v1.leases”…” type=io.containerd.grpc.v1
level=info msg=“loading plugin “io.containerd.grpc.v1.namespaces”…” type=io.containerd.grpc.v1
level=info msg=“loading plugin “io.containerd.internal.v1.opt”…” type=io.containerd.internal.v1
level=warning msg=“failed to load plugin io.containerd.internal.v1.opt” error=“mkdir /opt: read-only file system”
level=info msg=“loading plugin “io.containerd.grpc.v1.snapshots”…” type=io.containerd.grpc.v1
level=info msg=“loading plugin “io.containerd.grpc.v1.tasks”…” type=io.containerd.grpc.v1
level=info msg=“loading plugin “io.containerd.grpc.v1.version”…” type=io.containerd.grpc.v1
level=info msg=“loading plugin “io.containerd.grpc.v1.introspection”…” type=io.containerd.grpc.v1
level=info msg=serving… address=/var/run/balena-engine/containerd/balena-engine-containerd-debug.sock
level=info msg=serving… address=/var/run/balena-engine/containerd/balena-engine-containerd.sock
level=info msg=“containerd successfully booted in 0.003403s”
level=error msg=“AUFS was not found in /proc/filesystems” storage-driver=aufs
level=info msg=“stopping healthcheck following graceful shutdown” module=libcontainerd
level=info msg=“stopping event stream following graceful shutdown” error=“context canceled” module=libcontainerd namespace=plugins.moby
balenad[45104]: Error starting daemon: error initializing graphdriver: driver not supported
systemd[1]: balena.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: balena.service: Failed with result ‘exit-code’.
systemd[1]: Failed to start Balena Application Container Engine.

It failed to start balena because of aufs. I am running docker on windows and it uses overlay2. Can I run balena container on overlay2?

Thanks
Patrick

jimsynz · January 29, 2020, 8:39pm

Hi Patrick.

Recently Docker released new versions of Docker for Windows and Mac that no longer supports AUFS. Unfortunately some of our images still depend on AUFS because they predate the creation of the overlay and overlay2 drivers. We are actively working to migrate our images to support overlay2. The last version of Docker Desktop for Windows with AUFS support was 18.06.1-ce-win74 - until we have migrated all our images we suggest downgrading Docker Desktop or building inside an Ubuntu virtual machine which still supports AUFS.

Thanks,
James.

pli · January 31, 2020, 7:29pm

Hi @jimsynz ,

I tried on 18.06.1-ce-win74 as you suggested. journalctl --no-pager --unit balena.service shows

an 31 15:43:45 fe3f7bb7dca2 balenad[164]: time=“2020-01-31T15:43:45.458996257Z” level=info msg=“shim balena-engine-containerd-shim started” address=/containerd-shim/moby/34e84e313e6a906f1180677bc2f36a70ef6a3273deb71a402fb10b7066c9d992/shim.sock debug=false pid=47564
Jan 31 15:43:46 fe3f7bb7dca2 balenad[164]: time=“2020-01-31T15:43:46.133081177Z” level=info msg=“shim reaped” id=34e84e313e6a906f1180677bc2f36a70ef6a3273deb71a402fb10b7066c9d992
Jan 31 15:43:46 fe3f7bb7dca2 balenad[164]: time=“2020-01-31T15:43:46.142010722Z” level=info msg=“ignoring event” module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jan 31 15:46:35 fe3f7bb7dca2 balenad[164]: time=“2020-01-31T15:46:35.499070550Z” level=info msg=“shim balena-engine-containerd-shim started” address=/containerd-shim/moby/2f7e952e1a8d2c62f725fb63740e209c22021243edb7d6b3b908b89f77cf9004/shim.sock debug=false pid=47712

And balena ps shows no container is running.

Thanks

shaunmulligan · January 31, 2020, 7:43pm

Hi Patrick. have you tried using the latest version of the project, specifically the section on windows https://github.com/balena-os/balenaos-in-container#windows which also includes a migration script to convert the NUC image from AUFS to overlayFS, so the problems with AUFS drivers missing should not be a problem anymore either. Could you pull the latest from that repo and try run : PS> .\balenaos-in-container.ps1 -image resin/resinos:2.46.0_rev1.dev-intel-nuc -id test -c "$PWD\config.json"

pli · January 31, 2020, 11:58pm

Hi,

I found out my problem is caused by the aufs2overlay file. It is in dos format and it failed to run on linux. After I converted the file to unix format and everything is working.

Thanks

shaunmulligan · February 1, 2020, 12:05am

Thats great to hear and thanks for letting us know. Would you mind raising an issue on the repo of that project so we can get the fixed for future users

Topic		Replies	Views
Running Balena engine inside a container running on balena engine balenaEngine	18	1208	August 29, 2021
BalenaOS in Container on Google Compute Engine Help balenaOS	6	408	June 18, 2020
BalenaOS in a docker container does not connect to balenaCloud balenaOS docker , raspberrypi4	7	1232	May 5, 2021
Container access is only available for running containers balenaOS raspberrypi4	10	1339	October 29, 2020
Application container must be running for a terminal to be started. But the container is running! Product support	1	680	June 10, 2019

Running application on a docker

Related topics