Online(VPN Only) status - balena-engine not running

shmulik · May 16, 2024, 8:58am

Hi,

We have a Compulab IOT-Gate-IMX8Plus that runs balenaOS 5.1.9 and Supervisor Version 15.2.0
I moved this device to a new fleet and then I became aware of its ‘VPN Only’ status.
I have logged into the device, and it seems that the balena service is trying to activate but it keeps failing.
Executing ‘balena ps’ returned the following message: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?

I saw on other threads it can be disk space related, but it seems like there is enough space:

root@36325ff:~# df -h
Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        1.5G     0  1.5G   0% /dev
tmpfs                           1.8G   42M  1.8G   3% /run
/dev/mmcblk2p2                  451M  311M  113M  74% /mnt/sysroot/active
/dev/disk/by-state/resin-state   18M   82K   16M   1% /mnt/state
overlay                         451M  311M  113M  74% /
/dev/mmcblk2p6                   28G  4.1G   23G  16% /mnt/data
tmpfs                           1.8G     0  1.8G   0% /dev/shm
tmpfs                           4.0M     0  4.0M   0% /sys/fs/cgroup
tmpfs                           1.8G   88K  1.8G   1% /tmp
/dev/mmcblk2p1                   40M  1.9M   39M   5% /mnt/boot
tmpfs                           1.8G   56K  1.8G   1% /var/volatile


root@36325ff:~# df -i
Filesystem                      Inodes  IUsed   IFree IUse% Mounted on
devtmpfs                        368189    534  367655    1% /dev
tmpfs                           461477    957  460520    1% /run
/dev/mmcblk2p2                   60960   5599   55361   10% /mnt/sysroot/active
/dev/disk/by-state/resin-state    5112     81    5031    2% /mnt/state
overlay                          60960   5599   55361   10% /
/dev/mmcblk2p6                 2764800 142123 2622677    6% /mnt/data
tmpfs                           461477      1  461476    1% /dev/shm
tmpfs                             1024     17    1007    2% /sys/fs/cgroup
tmpfs                          1048576     18 1048558    1% /tmp
/dev/mmcblk2p1                       0      0       0     - /mnt/boot
tmpfs                           461477     57  461420    1% /var/volatile

Here are some more logs:

root@36325ff:~# systemctl status balena-engine
× balena.service - Balena Application Container Engine
     Loaded: loaded (/lib/systemd/system/balena.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/balena.service.d
             └─storagemigration.conf
     Active: failed (Result: exit-code) since Thu 2024-05-16 07:33:40 UTC; 571ms ago
TriggeredBy: × balena-engine.socket
       Docs: https://www.balena.io/docs/getting-started
    Process: 833600 ExecStart=/usr/bin/healthdog --healthcheck=${BALENAD_HEALTHDOG_HEALTHCHECK} /usr/bin/balenad $BALENAD_EXPERIMENTAL $BALENAD_DEBUG $BALENAD_LOGDRIVER $BALENAD_STORAGEDRIVER $BALENAD_HOST $BALENAD_DATA_ROOT $BALENAD_DELTA_STORAGEDRIVER $BALENAD_DELTA_DATA_ROOT $BALENAD_EXEC_ROOT $BALENAD_PIDFILE $BALENAD_DNS $BALENAD_BIP $BALENAD_CIDR $BALENAD_IPTABLES $BALENAD_MAX_DL_ATTEMPTS $BALENAD_CGROUPDRIVER $BALENAD_EXTRA_ARGS $BALENA_DEVELOPMENT_ARGS (code=exited, status=1/FAILURE)
   Main PID: 833600 (code=exited, status=1/FAILURE)
      Tasks: 76 (limit: 3451)
     Memory: 84.4M
     CGroup: /system.slice/balena.service
             ├─ 831269 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831268
             ├─ 831301 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831296
             ├─ 831343 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831342
             ├─ 831371 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831370
             ├─ 831397 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831396
             ├─ 831422 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831421
             ├─ 831455 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831450
             ├─ 831500 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831498
             ├─ 831528 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831527
             ├─ 831554 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831553
             ├─ 831579 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831578
             ├─ 831611 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831606
             ├─ 831654 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831653
             ├─ 831680 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831679
             ├─ 831704 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831703
             ├─ 831729 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831728
             ├─ 831762 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831757
             ├─ 831805 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831804
             ├─ 831833 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831832
             ├─ 831858 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831857
             ├─ 831883 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831882
             ├─ 831916 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831911
             ├─ 831958 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831956
             ├─ 831985 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 831984
             ├─ 832010 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832009
             ├─ 832035 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832034
             ├─ 832067 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832062
             ├─ 832111 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832109
             ├─ 832138 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832137
             ├─ 832164 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832163
             ├─ 832190 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832189
             ├─ 832222 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832217
             ├─ 832265 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832264
             ├─ 832293 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832292
             ├─ 832318 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832317
             ├─ 832343 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832342
             ├─ 832387 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832382
             ├─ 832430 /proc/self/exe --healthcheck /usr/lib/balena/balena-healthcheck --pid 832428

....

These logs are after I tried to start balena service using systemctl start balena:

root@36325ff:~# journalctl -xeu balena.service
May 16 07:46:38 36325ff balenad[843774]: time="2024-05-16T07:46:38.454338026Z" level=info msg="Starting up"
May 16 07:46:38 36325ff balenad[843774]: time="2024-05-16T07:46:38.454454157Z" level=warning msg="Running experimental build"
May 16 07:46:38 36325ff balenad[841410]: Parent process exited
May 16 07:46:38 36325ff balenad[843774]: time="2024-05-16T07:46:38.458789990Z" level=info msg="libcontainerd: started new balena-engine-containerd process" pid=843786
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38Z" level=warning msg="containerd config version `1` has been deprecated and will be removed in containerd v2.0, please switch to version `2`, >
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.571160382Z" level=info msg="starting containerd" revision= version=1.6.6+unknown
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.653745721Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.653941355Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.657171510Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed:>
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.657302266Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.657427398Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.657957798Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.658060928Z" level=info msg="metadata content store policy set" policy=shared
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.658425320Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.658525325Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.658597079Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.658700709Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.658762712Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.658825090Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.658883092Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.658944720Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.659006973Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.659066226Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.659127854Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.659184232Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.659640754Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.659950894Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.661433090Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.661591597Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.661665476Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.661856110Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.661927113Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.661994867Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662056370Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662119623Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662182501Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662261504Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662326383Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662396261Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662458389Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662523642Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662612271Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662699526Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662765029Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662848158Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.662974289Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." type=io.containerd.ttrpc.v1
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.663052792Z" level=warning msg="failed to load plugin io.containerd.ttrpc.v1.task" error="no io.containerd.event.v1 plugins with id publisher: not found"
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.663649821Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd-debug.sock
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.663917959Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd.sock.ttrpc
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.664128719Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd.sock
May 16 07:46:38 36325ff balenad[843786]: time="2024-05-16T07:46:38.664212348Z" level=info msg="containerd successfully booted in 0.096683s"

root@36325ff:~# journalctl -xe
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.152028053Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.152117807Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.152175060Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.152311566Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.152389195Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.152454323Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.152514576Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.152579204Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.152640957Z" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.152702960Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.152765088Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.152827466Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.153106979Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.153279363Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.154451794Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.154593426Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.154664179Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.154841188Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.154901940Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.154963443Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155020446Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155076699Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155133327Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155189079Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155250207Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155325461Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155387339Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155480468Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155545221Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155615725Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155693603Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155767482Z" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155890863Z" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." type=io.containerd.ttrpc.v1
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.155973367Z" level=warning msg="failed to load plugin io.containerd.ttrpc.v1.task" error="no io.containerd.event.v1 plugins with id publisher: not found"
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.156624523Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd-debug. Sock
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.156898661Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd.sock.ttrpc
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.157136798Z" level=info msg=serving... address=/var/run/balena-engine/containerd/balena-engine-containerd.sock
May 16 07:47:16 36325ff balenad[844270]: time="2024-05-16T07:47:16.157229927Z" level=info msg="containerd successfully booted in 0.099308s"
May 16 07:47:16 36325ff balenad[844258]: time="2024-05-16T07:47:16.191754584Z" level=info msg="Storage migration skipped: %!s(<nil>)"
May 16 07:47:16 36325ff balenad[844258]: time="2024-05-16T07:47:16.223224844Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugin.moby
May 16 07:47:16 36325ff kernel: EXT4-fs error (device mmcblk2p6): ext4_lookup:1833: inode #1388564: comm balenad: deleted inode referenced: 1784793
May 16 07:47:16 36325ff balenad[844258]: time="2024-05-16T07:47:16.223345350Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
May 16 07:47:17 36325ff balenad[841924]: Parent process exited
May 16 07:47:17 36325ff balenad[844258]: failed to start daemon: error initializing graphdriver: lstat /var/lib/docker/overlay2/opaque-bug-check446572362: structure needs cleaning
May 16 07:47:17 36325ff systemd[1]: balena.service: Main process exited, code=exited, status=1/FAILURE
May 16 07:47:17 36325ff systemd[1]: balena.service: Failed with result 'exit-code'.
May 16 07:47:17 36325ff systemd[1]: Failed to start Balena Application Container Engine.
May 16 07:47:17 36325ff systemd[1]: balena.service: Start request repeated too quickly.
May 16 07:47:17 36325ff systemd[1]: balena.service: Failed with result 'exit-code'.
May 16 07:47:17 36325ff systemd[1]: Failed to start Balena Application Container Engine.
May 16 07:47:17 36325ff systemd[1]: balena-engine.socket: Failed with result 'service-start-limit-hit'

Thanks,
Shmulik

mpous · May 16, 2024, 10:10am

Hello @shmulik first of all welcome to the balena community!

It looks like you have an issue with the balenaEngine running on the device.

Could you please share the logs from ls -la /run | grep 'balena-engine'?

However i’m not sure about this

May 16 07:47:17 36325ff balenad[844258]: failed to start daemon: error initializing graphdriver: lstat /var/lib/docker/overlay2/opaque-bug-check446572362: structure needs cleaning
May 16 07:47:17 36325ff systemd[1]: balena.service: Main process exited, code=exited, status=1/FAILURE

Let’s stay connected!

shmulik · May 16, 2024, 10:39am

Hello Marc, Thank you for the quick response!
Here are the requested logs:

root@36325ff:~# ls -la /run | grep 'balena-engine'
drwx------  4 root   root            80 May 16 10:32 balena-engine
-rw-r--r--  1 root   root             6 May 16 10:32 balena-engine.pid
srw-rw----  1 root   balena-engine    0 May 16 10:32 balena-engine.sock
srw-rw----  1 root   balena-engine    0 May 15 13:51 balena-host.sock
lrwxrwxrwx  1 root   root            22 May 15 13:51 balena.pid -> /run/balena-engine.pid
srw-rw----  1 root   balena-engine    0 May 16 10:32 balena.sock
lrwxrwxrwx  1 root   root            22 May 15 13:51 docker.pid -> /run/balena-engine.pid

Thanks,
Shmulik

mpous · May 16, 2024, 10:55am

@shmulik did this work before?

shmulik · May 16, 2024, 11:17am

@mpous Yes, it used to work properly, but I can’t tell when it changed to ‘VPN Only’.