Preload fail: driver not supported

We’ve been trying to use the balena preload command for this project.

Yesterday me managed to get it working no problem. But today we started getting an error on different Linux machines.

This is the used command:

balena preload balena-cloud-myFleet-raspberrypi3-64-2.80.3+rev1-v12.7.0.img --fleet myFleet

And these are the logs we get:

Building Docker preloader image. [========================] 100%
| Checking that the image is a writable file
| Finding a free tcp port
| Checking if the image is an edison zip archive
- Creating preloader container
- Starting preloader container
\ Fetching application gh_luandro/guarani
\ Reading image information
| Cleaning up temporary files
An error has occurred executing internal preload command 'get_image_info':
{"command":"get_image_info","parameters":{}}

Status code: 1
Error: Traceback (most recent call last):
  File "/usr/src/app/preload.py", line 973, in <module>
    result = method(**data.get("parameters", {}))
  File "/usr/src/app/preload.py", line 924, in get_image_info
    get_images_and_supervisor_version()
  File "/usr/src/app/preload.py", line 785, in get_images_and_supervisor_version
    return _get_images_and_supervisor_version()
  File "/usr/src/app/preload.py", line 745, in _get_images_and_supervisor_version
    with docker_context_manager(driver, mountpoint):
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/src/app/preload.py", line 586, in docker_context_manager
    running_dockerd = start_docker_daemon(storage_driver, docker_dir)
  File "/usr/src/app/preload.py", line 555, in start_docker_daemon
    running_dockerd.wait()
  File "/usr/lib/python3.8/site-packages/sh.py", line 792, in wait
    self.handle_command_exit_code(exit_code)
  File "/usr/lib/python3.8/site-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_1: 

  RAN: /usr/bin/dockerd --storage-driver=aufs --data-root=/tmp/tmpn6lq50ue/docker --tls=false --host=tcp://0.0.0.0:40631

  STDOUT:


  STDERR:
time="2021-10-20T23:50:39.368684792Z" level=info msg="Starting up"
time="2021-10-20T23:50:39.369157340Z" level=warning msg="Binding to IP address without --tlsverify is insecure and gives root access on this machine to everyone who has access to your network." host="tcp://0.0.0.0:40631"
time="2021-10-20T23:50:39.369184447Z" level=warning msg="Binding to an IP address, even on localhost, can also give access to scripts run in a browser. Be safe out there!" host="tcp://0.0.0.0:40631"
time="2021-10-20T23:50:40.370484650Z" level=info msg="libcontainerd: started new containerd process" pid=167
time="2021-10-20T23:50:40.370579085Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2021-10-20T23:50:40.370594400Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2021-10-20T23:50:40.370625005Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2021-10-20T23:50:40.370645758Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2021-10-20T23:50:40.393816201Z" level=info msg="starting containerd" revision=05f951a3781f4f2c1911b05e61c160e9c30eaa8e version=v1.4.4
time="2021-10-20T23:50:40.446314735Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2021-10-20T23:50:40.446462551Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2021-10-20T23:50:40.447419052Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: can't change directory to '/lib/modules': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2021-10-20T23:50:40.447468525Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2021-10-20T23:50:40.447690303Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /tmp/tmpn6lq50ue/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2021-10-20T23:50:40.447724638Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2021-10-20T23:50:40.447767215Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2021-10-20T23:50:40.447786863Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2021-10-20T23:50:40.447910443Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2021-10-20T23:50:40.448056571Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2021-10-20T23:50:40.448205511Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /tmp/tmpn6lq50ue/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2021-10-20T23:50:40.448227028Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2021-10-20T23:50:40.448263817Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2021-10-20T23:50:40.448281059Z" level=info msg="metadata content store policy set" policy=shared
time="2021-10-20T23:50:40.448490241Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
time="2021-10-20T23:50:40.448516907Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
time="2021-10-20T23:50:40.448566871Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
time="2021-10-20T23:50:40.448629314Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
time="2021-10-20T23:50:40.448655855Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
time="2021-10-20T23:50:40.448680393Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
time="2021-10-20T23:50:40.448704887Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
time="2021-10-20T23:50:40.448727542Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
time="2021-10-20T23:50:40.448751859Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
time="2021-10-20T23:50:40.448784771Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
time="2021-10-20T23:50:40.448808505Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
time="2021-10-20T23:50:40.449019622Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
time="2021-10-20T23:50:40.449175385Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
time="2021-10-20T23:50:40.449493412Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
time="2021-10-20T23:50:40.449532420Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
time="2021-10-20T23:50:40.449594109Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
time="2021-10-20T23:50:40.449619914Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
time="2021-10-20T23:50:40.449643458Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
time="2021-10-20T23:50:40.449665357Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
time="2021-10-20T23:50:40.449687113Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
time="2021-10-20T23:50:40.449712369Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
time="2021-10-20T23:50:40.449734969Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
time="2021-10-20T23:50:40.449756889Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
time="2021-10-20T23:50:40.449779218Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
time="2021-10-20T23:50:40.450124919Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
time="2021-10-20T23:50:40.450156792Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
time="2021-10-20T23:50:40.450177987Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
time="2021-10-20T23:50:40.450196598Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
time="2021-10-20T23:50:40.450562308Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
time="2021-10-20T23:50:40.450662355Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
time="2021-10-20T23:50:40.450752549Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
time="2021-10-20T23:50:40.450774321Z" level=info msg="containerd successfully booted in 0.057769s"
time="2021-10-20T23:50:40.461049597Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2021-10-20T23:50:40.461092515Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2021-10-20T23:50:40.461119301Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2021-10-20T23:50:40.461148889Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2021-10-20T23:50:40.469097722Z" level=info msg="parsed scheme: \"unix\"" module=grpc
time="2021-10-20T23:50:40.469121171Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
time="2021-10-20T23:50:40.469148342Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var/run/docker/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
time="2021-10-20T23:50:40.469164003Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
time="2021-10-20T23:50:40.470031187Z" level=warning msg="[graphdriver] WARNING: the aufs storage-driver is deprecated, and will be removed in a future release"
time="2021-10-20T23:50:40.471098757Z" level=error msg="AUFS was not found in /proc/filesystems" storage-driver=aufs
time="2021-10-20T23:50:40.471761613Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
time="2021-10-20T23:50:40.471783814Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: error initializing graphdriver: driver not supported




Additional information may be available with the `--debug` flag.

For further help or support, visit:
https://www.balena.io/docs/reference/balena-cli/#support-faq-and-troubleshooting

Why does Balena try to use --storage-driver=aufs is Docker doesn’t use it anymore?

Found some answers from @klutchell on this issue:

At the moment there are some device types that are using aufs on balenaOS (notably the Raspberry Pi range). We are transitioning all of them to overlay2, likely in one of the soon upcoming releases

What confuses me is how yesterday we managed to get it working, and today we get an error… :confused:

Hey @luandro, the following errors indicate that the host OS of the workstation does not support AUFS.

/usr/bin/dockerd --storage-driver=aufs --data-root=/tmp/tmpn6lq50ue/docker --tls=false --host=tcp://0.0.0.0:40631
failed to start daemon: error initializing graphdriver: driver not supported

Since you’re running on linux you can see your supported filesystems by running cat /proc/filesystems and looking for overlay and aufs. Recently we’ve seen aufs getting removed from a lot of recent kernels so it’s possible a system update has removed support when it may have previously been present.

Can you confirm that you don’t have the same issue attempting to preload Raspberry Pi 4? Asking because the Pi4 doesn’t use AUFS.