Error preloading BalenaOS

Product support
1

I am preloading a bunch of images. The following are working (all tested with latest production OS version pulled just before posting):

Orange pi zero (armv7hf fleet)
Raspberry pi 4 (aarch64 fleet)

The following are reporting an error:

Balena Fin (armv7hf fleet)
Raspberry pi 3 64 bit (aarch64 fleet
Raspberry pi 2 (armv7hf fleet)
Raspberry pi zero (armv6hf fleet)
Raspberry pi 3 (armv7 fleet

I don’t think it is related to switching away from Balena images but noting just in case: What do the balenalib images do?

I have been running processes in a GitHub workflow using Ubuntu and have seen: Preload fails with "error initializing graphdriver: driver not supported" · Issue #1099 · balena-io/balena-cli · GitHub

balena --version
12.51.1

Building Docker preloader image. [========================] 100%

| Checking that the image is a writable file

| Finding a free tcp port

| Checking if the image is an edison zip archive

/ Creating preloader container

- Starting preloader container

| Fetching application

- Reading image information

\ Cleaning up temporary files

An error has occurred executing internal preload command 'get_image_info':

{"command":"get_image_info","parameters":{}}

Status code: 1

Error: Traceback (most recent call last):

File "/usr/src/app/preload.py", line 973, in <module>

result = method(**data.get("parameters", {}))

File "/usr/src/app/preload.py", line 924, in get_image_info

get_images_and_supervisor_version()

File "/usr/src/app/preload.py", line 785, in get_images_and_supervisor_version

return _get_images_and_supervisor_version()

File "/usr/src/app/preload.py", line 745, in _get_images_and_supervisor_version

with docker_context_manager(driver, mountpoint):

File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__

return next(self.gen)

File "/usr/src/app/preload.py", line 586, in docker_context_manager

running_dockerd = start_docker_daemon(storage_driver, docker_dir)

File "/usr/src/app/preload.py", line 555, in start_docker_daemon

running_dockerd.wait()

File "/usr/lib/python3.8/site-packages/sh.py", line 792, in wait

self.handle_command_exit_code(exit_code)

File "/usr/lib/python3.8/site-packages/sh.py", line 815, in handle_command_exit_code

raise exc

sh.ErrorReturnCode_1:

RAN: /usr/bin/dockerd --storage-driver=aufs --data-root=/tmp/tmpgsj7wkij/docker --tls=false --host=tcp://0.0.0.0:63787

STDOUT:

STDERR:

time="2021-11-05T12:16:06.712693590Z" level=info msg="Starting up"

time="2021-11-05T12:16:06.713260257Z" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://0.0.0.0:63787"

time="2021-11-05T12:16:06.713280882Z" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://0.0.0.0:63787"

time="2021-11-05T12:16:07.716799966Z" level=info msg="libcontainerd: started new containerd process" pid=187

time="2021-11-05T12:16:07.716908049Z" level=info msg="parsed scheme: \"unix\"" module=grpc

time="2021-11-05T12:16:07.716918549Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc

time="2021-11-05T12:16:07.716954716Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc

time="2021-11-05T12:16:07.716966716Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc

time="2021-11-05T12:16:07.750452257Z" level=info msg="starting containerd" revision=05f951a3781f4f2c1911b05e61c160e9c30eaa8e version=v1.4.4

time="2021-11-05T12:16:07.766339424Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1

time="2021-11-05T12:16:07.766679549Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1

time="2021-11-05T12:16:07.768653257Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: can't change directory to '/lib/modules': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1

time="2021-11-05T12:16:07.768690841Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1

time="2021-11-05T12:16:07.768957216Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /tmp/tmpgsj7wkij/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1

time="2021-11-05T12:16:07.769036216Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1

time="2021-11-05T12:16:07.769067007Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"

time="2021-11-05T12:16:07.769075591Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1

time="2021-11-05T12:16:07.773150007Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1

time="2021-11-05T12:16:07.780166841Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1

time="2021-11-05T12:16:07.780685924Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /tmp/tmpgsj7wkij/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1

time="2021-11-05T12:16:07.780706549Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1

time="2021-11-05T12:16:07.780760424Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"

time="2021-11-05T12:16:07.780771424Z" level=info msg="metadata content store policy set" policy=shared

time="2021-11-05T12:16:07.783178299Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1

time="2021-11-05T12:16:07.783204466Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1

time="2021-11-05T12:16:07.783258841Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1

time="2021-11-05T12:16:07.783288216Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1

time="2021-11-05T12:16:07.783295549Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1

time="2021-11-05T12:16:07.783303632Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1

time="2021-11-05T12:16:07.783312007Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1

time="2021-11-05T12:16:07.783318299Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1

time="2021-11-05T12:16:07.783325132Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1

time="2021-11-05T12:16:07.783332632Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1

time="2021-11-05T12:16:07.783339799Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1

time="2021-11-05T12:16:07.783565341Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2

time="2021-11-05T12:16:07.783687299Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1

time="2021-11-05T12:16:07.784016799Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1

time="2021-11-05T12:16:07.784030882Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1

time="2021-11-05T12:16:07.784072132Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1

time="2021-11-05T12:16:07.784079799Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1

time="2021-11-05T12:16:07.784089716Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1

time="2021-11-05T12:16:07.784354841Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1

time="2021-11-05T12:16:07.784365882Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1

time="2021-11-05T12:16:07.784379799Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1

time="2021-11-05T12:16:07.784386299Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1

time="2021-11-05T12:16:07.784393007Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1

time="2021-11-05T12:16:07.784400591Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1

time="2021-11-05T12:16:07.784718091Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1

time="2021-11-05T12:16:07.784728507Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1

time="2021-11-05T12:16:07.784734966Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1

time="2021-11-05T12:16:07.784740466Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1

time="2021-11-05T12:16:07.785046591Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock

time="2021-11-05T12:16:07.785272466Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc

time="2021-11-05T12:16:07.785345466Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock

time="2021-11-05T12:16:07.785357966Z" level=info msg="containerd successfully booted in 0.039506s"

time="2021-11-05T12:16:07.804042382Z" level=info msg="parsed scheme: \"unix\"" module=grpc

time="2021-11-05T12:16:07.804170174Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc

time="2021-11-05T12:16:07.804192049Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc

time="2021-11-05T12:16:07.804201466Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc

time="2021-11-05T12:16:07.805829799Z" level=info msg="parsed scheme: \"unix\"" module=grpc

time="2021-11-05T12:16:07.805909674Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc

time="2021-11-05T12:16:07.805925132Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc

time="2021-11-05T12:16:07.805930424Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc

time="2021-11-05T12:16:07.807649966Z" level=warning msg="[graphdriver] WARNING: the aufs storage-driver is deprecated, and will be removed in a future release"

time="2021-11-05T12:16:07.810269841Z" level=error msg="AUFS was not found in /proc/filesystems" storage-driver=aufs

time="2021-11-05T12:16:07.810869716Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby

time="2021-11-05T12:16:07.810957799Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd

time="2021-11-05T12:16:08.812722258Z" level=warning msg="grpc: addrConn.createTransport failed to connect to {unix:///var/run/docker/containerd/containerd.sock <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing dial unix:///var/run/docker/containerd/containerd.sock: timeout\". Reconnecting..." module=grpc

failed to start daemon: error initializing graphdriver: driver not supported
3

Thanks @maggie0002 for reaching out. A couple of clarifications on the above:

Regards,
N

4

Yes from a workflow using GitHub runners.

This could be it. GitHub bumped docker on their runners to 20.10.10

Docker change log for that release shows they dropped aufs: Docker Engine release notes | Docker Documentation

What’s the Balena plan for overlay? I read on GitHub the migration strategy to overlay is ongoing, but the post was raised 2 and a half years ago and it’s gradually meaning there aren’t many images left Balena supports.

The Linux vm option will now mean rolling back docker before 20.10.10, far from ideal.

It would be helpful to see a list of which images are overlay and which I should try to avoid, is that available?

5

Hi Maggie, the aufs to overlay migration takes place with balenaOS 2.84.0, so it will happen as device types get updated. The engine will then migrate applications from aufs to overlay avoiding the need to re-download containers.

The affected device types are:

beaglebone-black
intel-edison
intel-nuc
odroid-c1
raspberry-pi
raspberry-pi2
raspberrypi3
raspberrypi3-64
fincm3
revpi-core-3
npe-x500-m3
up-board

Hope that helps.

1 Like
7

Really helpful thanks. Surprising the raspberry pi’s and fin are the exceptions as they are such popular boards and your own boards, I had figured it would be the other way around. Seems like the vast majority are ready to go then and will await the others. Bit confused as to why orange pi zero works at BalenaOS 2.53.9+rev1 but it helps a bit to decipher the versions.

8

Hi again, it only depends on the time when the boards were introduced, older boards are more likely to be aufs. As to the orange pi zero, it’s not in the list above so it’s an overlay board. It’s not part of the RaspberryPi device repository but of the allwinner family, and all those are overlay2 based.

1 Like
10

Is there any way to work with the later versions like 2.84 with raspberry pi zero before it’s made generally available? Seems like it’s been stuck at 2.83 and not sure how long it would take to update the prod versions (or not even sure why that’s necessary if it’s passing tests)?

11

I have the same issue with raspberrypi3-64 but I don’t understand how to solve it from the comments in this thread.

Can anyone please clarify?

Will using an image < 2.84.0 fix this?

12

Using the latest images should fix it. It is working for me now for RPI3-64

13

I checked again and figured out that I’m actually not using the raspberrypi3-64 image, but the raspberry-pi image (because it’s a Zero W). The balenaOS version is v2.83.21+rev1, which is recommended for the device (and the latest version available).

In the meanwhile I found this discussion and the related PR:

So I’m gonna try this approach and install an older docker version in a VM.

14

Mkay. It seems to work now with Ubuntu in a VM. But it doesn’t make any sense at all, because my host system is Ubuntu as well. The only difference is that I went with 20.04 LTS for the VM while the host is 21.10. balena-cli and docker are the same versions I used before on the host, because I found no way to install a Docker version as old as it was recommended in the discussion. Hm, so maybe it is actually the older OS that did the trick or just pure magic… Who knows …

15

Hi, I suspect the problem is with Ubuntu 21.10 defaulting to using cgroups2. You might want to try booting with systemd.unified_cgroup_hierarchy=0 in the kernel command line see if it makes a difference.