Issue failed to start daemon

Product support balenaOS
1

Hardware & OS version

  • Raspberry Pi 4
  • balenaOS 2.95.8

Problem

The balena-engine won’t start any more reporting no space left on device.

Logs from device systemctl:

Sep 05 14:02:06 22df74e balenad[2457]: failed to start daemon: Unable to get the TempDir under /var/lib/docker: mkdir /var/lib/docker/tmp: no space left on device
Sep 05 14:02:06 22df74e systemd[1]: balena.service: Main process exited, code=exited, status=1/FAILURE
Sep 05 14:02:06 22df74e systemd[1]: balena.service: Failed with result 'exit-code'.
Sep 05 14:02:06 22df74e systemd[1]: Failed to start Balena Application Container Engine.
Sep 05 14:02:07 22df74e balena-supervisor[2504]: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
Sep 05 14:02:07 22df74e balenad[2505]: Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
Sep 05 14:02:07 22df74e systemd[1]: balena.service: Found left-over process 1558 (exe) in control group while starting unit. Ignoring.

 df -h
Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        345M     0  345M   0% /dev
tmpfs                           480M   22M  459M   5% /run
/dev/mmcblk0p2                  300M  273M  7.0M  98% /mnt/sysroot/active
/dev/disk/by-state/resin-state   18M  403K   16M   3% /mnt/state
overlay                         300M  273M  7.0M  98% /
/dev/mmcblk0p6                   28G   12G   15G  45% /mnt/data
tmpfs                           480M     0  480M   0% /dev/shm
tmpfs                           4.0M     0  4.0M   0% /sys/fs/cgroup
tmpfs                           480M     0  480M   0% /tmp
/dev/mmcblk0p1                   40M  7.2M   33M  19% /mnt/boot
tmpfs                           480M   32K  480M   1% /var/volatile

Only a few of devices in our fleet have this issue.

Thanks!

2

Hello @benjboy ,

First of all, is there any particular reason why you are using a very outdated version of Balena OS 2.95.8 ?

Firat

3

We are using openBalena so updating os is not an easy option.

4

ok so,

It seems your Balena engine isn’t starting because the root filesystem is almost full.

I think, the engine is trying to create a temporary directory under /var/lib/docker, but there’s no more free space on the system partition.

Can you please try to moving the Balena/Docker data directory (/var/lib/docker or /var/lib/balena) to the larger /mnt/data partition and pointing the engine there.

Once the data root is moved, make sure to free up some space on the system partition, then try to restart the Balena service.

–

Firat