No space left on device while downloading container

After upgrading BalenaOS from 2.58.0 to 2.83.21 the device can’t download and install the container.

My storage is a 4GB eMMC. The images size is about 650MB.

df -h

Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        148M     0  148M   0% /dev
tmpfs                           216M  4.0K  216M   1% /tmp
/dev/mmcblk1p3                  307M  186M  102M  65% /mnt/sysroot/active
/dev/disk/by-state/resin-state   19M  226K   17M   2% /mnt/state
overlay                         307M  186M  102M  65% /
/dev/mmcblk1p6                  170M   70M   88M  45% /mnt/data
tmpfs                           216M     0  216M   0% /dev/shm
tmpfs                           216M  4.8M  211M   3% /run
tmpfs                           216M     0  216M   0% /sys/fs/cgroup
/dev/mmcblk1p2                   40M  9.6M   30M  25% /mnt/boot
tmpfs                           216M   28K  216M   1% /var/volatile

fdisk -l

Device           Start     End Sectors  Size Type
/dev/mmcblk1p1    1024    5119    4096    2M Linux reserved
/dev/mmcblk1p2    5120   87039   81920   40M Linux filesystem
/dev/mmcblk1p3   87040  742399  655360  320M Linux reserved
/dev/mmcblk1p4  742400 1397759  655360  320M Linux reserved
/dev/mmcblk1p5 1397760 1438719   40960   20M Linux reserved
/dev/mmcblk1p6 1438720 7732223 6293504    3G Linux reserved

The error I get in BalenaCloud log:

Failed to download image 'registry2.balena-cloud.com/v2/xxxxxxxxxxxxxxxxxxxxxx@sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx' due to 'failed to register layer: Error processing tar file(exit status 1): write /usr/sbin/genl: no space left on device'

journalctl -a -f -u balena-supervisor

Sep 07 12:41:26 ecddf19 balena-supervisor[1228]: [event]   Event: Image download error {"error":{"message":"failed to register layer: Error processing tar file(exit status 1): write /usr/sbin/genl: no space left on device","stack":"Error: failed to register layer: Error processing tar file(exit status 1): write /usr/sbin/genl: no space left on device\n    at Stream.<anonymous> (/usr/src/app/dist/app.js:10:2309272)\n    at Stream.emit (events.js:310:20)\n    at drain (/usr/src/app/dist/app.js:2:306286)\n    at Stream.stream.queue.stream.push (/usr/src/app/dist/app.js:2:306693)\n    at Parser.parser.onToken (/usr/src/app/dist/app.js:10:256101)\n    at Parser.proto.write (/usr/src/app/dist/app.js:10:704892)\n    at Stream.<anonymous> (/usr/src/app/dist/app.js:10:254565)\n    at Stream.stream.write (/usr/src/app/dist/app.js:2:306562)\n    at IncomingMessage.ondata (_stream_readable.js:695:22)\n    at IncomingMessage.emit (events.js:310:20)\n    at addChunk (_stream_readable.js:286:12)\n    at readableAddChunk (_stream_readable.js:268:9)\n    at IncomingMessage.Readable.push (_stream_readable.js:209:10)\n    at HTTPParser.parserOnBody (_http_common.js:132:24)\n    at Socket.socketOnData (_http_client.js:476:22)\n    at Socket.emit (events.js:310:20)"},"image":{"name":"registry2.balena-cloud.com/v2/xxxxxxxxxxxxxxxxxxxxxx@sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","appId":XXXXXXX,"serviceId":XXXXXX,"serviceName":"main","imageId":XXXXXXX,"releaseId":XXXXXX,"dependent":0}}
Sep 07 12:41:28 ecddf19 balena-supervisor[1228]: [error]   Scheduling another update attempt in 16000ms due to failure:  Error: Failed to apply state transition steps.  Steps:["fetch"]
Sep 07 12:41:28 ecddf19 balena-supervisor[1228]: [error]         at fn (/usr/src/app/dist/app.js:6:8594)
Sep 07 12:41:28 ecddf19 balena-supervisor[1228]: [error]   Device state apply error Error: Failed to apply state transition steps.  Steps:["fetch"]
Sep 07 12:41:28 ecddf19 balena-supervisor[1228]: [error]         at fn (/usr/src/app/dist/app.js:6:8594)

I have also purged data from within BalenaCloud with no change,

The device is a STM32MP1 based board so maybe I have to do some more changes in my meta-layer. I have notices some renaming from resin to balena between the versions and also that the rootA and rootB size is increased from 300MB to 312MB.

Hello @Ankan sorry to hear this. I contacted the balenaOS team to see how we can help you with this.

Let’s stay connected

The data partition on this device appears to only be 170M, with 88M available

/dev/mmcblk1p6                  170M   70M   88M  45% /mnt/data
  1. How big is the image you are trying to download?
  2. Can we confirm the data partition is not mounted incorrectly as read-only via the mount command?
  3. Can we confirm the paths the engine is using with balena info?

Thanks!

  1. The image is about 650MB
  2. mount output:
    /dev/mmcblk1p6 on /resin-data type ext4 (rw,relatime)
  3. balena info output: Docker Root Dir: /var/lib/docker

Here is the output from df -h on same device running BalenaOS 2.58.0 (that is working)

Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        148M     0  148M   0% /dev
tmpfs                           216M     0  216M   0% /tmp
/dev/mmcblk1p3                  300M  189M   92M  68% /mnt/sysroot/active
/dev/disk/by-state/resin-state   19M  317K   17M   2% /mnt/state
overlay                         300M  189M   92M  68% /
tmpfs                           216M     0  216M   0% /dev/shm
tmpfs                           216M  992K  215M   1% /run
tmpfs                           216M     0  216M   0% /sys/fs/cgroup
/dev/mmcblk1p2                   40M  9.8M   30M  25% /mnt/boot
tmpfs                           216M   20K  216M   1% /var/volatile
/dev/mmcblk1p6                  3.0G 1014M  1.9G  36% /mnt/data
/dev/mmcblk1p4                  300M  2.1M  278M   1% /mnt/sysroot/inactive

As notices by @klutchell, the size of the mount of /mnt/data is small while the partition /dev/mmcblk1p6 is big.

Can the issue come from the change in 2.59.0:

  • resin-filesystem-expand: Omit fs check and resize if partition is mounted [Alex Gonzalez]
  • initrdscripts: Expand the resin-data filesystem [Alex Gonzalez]

Got below result with journalctl | grep "mount"

Sep 06 12:00:04 localhost kernel: EXT4-fs (mmcblk1p3): mounted filesystem with ordered data mode. Opts: (null)
Sep 06 12:00:04 localhost kernel: EXT4-fs (mmcblk1p5): mounted filesystem with ordered data mode. Opts: (null)
Sep 06 12:00:04 localhost kernel: EXT4-fs (mmcblk1p3): re-mounted. Opts: (null)
Sep 06 12:00:04 localhost kernel: EXT4-fs (mmcblk1p6): mounted filesystem with ordered data mode. Opts: (null)
Sep 06 12:00:04 localhost resin-partition-mounter[469]: INFO: /dev/disk/by-state/resin-state (resin-state) already mounted in /mnt/state.
Sep 06 12:00:04 localhost resin-partition-mounter[471]: INFO: Mounting /dev/disk/by-state/resin-boot (resin-boot) in /mnt/boot.
Sep 06 12:00:04 localhost resin-partition-mounter[481]: INFO: /dev/disk/by-state/active (active) already mounted in /mnt/sysroot/active.
Sep 06 12:00:04 localhost resin-partition-mounter[576]: INFO: /dev/disk/by-state/resin-data (resin-data) already mounted in /mnt/data.
Sep 06 12:00:04 localhost bindmount[598]: INFO: Bindmounting /etc/openvpn in /mnt/state/root-overlay/etc/openvpn ...
Sep 06 12:00:04 localhost bindmount[598]: INFO: Successfully mounted /mnt/state/root-overlay/etc/openvpn.
Sep 06 12:00:04 localhost bindmount[599]: INFO: Bindmounting /etc/NetworkManager/system-connections in /mnt/state/root-overlay/etc/NetworkManager/system-connections ...
Sep 06 12:00:04 localhost bindmount[599]: INFO: Successfully mounted /mnt/state/root-overlay/etc/NetworkManager/system-connections.
Sep 06 12:00:04 localhost bindmount[600]: INFO: Bindmounting /home/root/.docker in /mnt/state/root-overlay/home/root/.docker ...
Sep 06 12:00:04 localhost bindmount[600]: INFO: Successfully mounted /mnt/state/root-overlay/home/root/.docker.
Sep 06 12:00:04 localhost bindmount[601]: INFO: Bindmounting /var/lib/systemd in /mnt/state/root-overlay/var/lib/systemd ...
Sep 06 12:00:04 localhost bindmount[601]: WARN: /var/lib/systemd is not an empty entry. You are going to shadow content.
Sep 06 12:00:04 localhost bindmount[601]: INFO: Successfully mounted /mnt/state/root-overlay/var/lib/systemd.
Sep 06 12:00:04 localhost bindmount[602]: INFO: Bindmounting /var/lib/NetworkManager in /mnt/state/root-overlay/var/lib/NetworkManager ...
Sep 06 12:00:04 localhost bindmount[602]: INFO: Successfully mounted /mnt/state/root-overlay/var/lib/NetworkManager.
Sep 06 12:00:04 localhost bindmount[603]: INFO: Bindmounting /var/lib/bluetooth in /mnt/state/root-overlay/var/lib/bluetooth ...
Sep 06 12:00:04 localhost bindmount[603]: INFO: Successfully mounted /mnt/state/root-overlay/var/lib/bluetooth.
Sep 06 12:00:04 localhost bindmount[604]: INFO: Bindmounting /usr/share/ca-certificates/balena in /mnt/state/root-overlay/usr/share/ca-certificates/balena ...
Sep 06 12:00:04 localhost bindmount[604]: INFO: Successfully mounted /mnt/state/root-overlay/usr/share/ca-certificates/balena.
Sep 06 12:00:04 localhost bindmount[606]: INFO: Bindmounting /etc/NetworkManager/conf.d in /mnt/state/root-overlay/etc/NetworkManager/conf.d ...
Sep 06 12:00:04 localhost bindmount[606]: INFO: Successfully mounted /mnt/state/root-overlay/etc/NetworkManager/conf.d.
Sep 06 12:00:04 localhost bindmount[607]: INFO: Bindmounting /etc/balena-supervisor in /mnt/state/root-overlay/etc/balena-supervisor ...
Sep 06 12:00:04 localhost bindmount[607]: WARN: /etc/balena-supervisor is not an empty entry. You are going to shadow content.
Sep 06 12:00:04 localhost bindmount[607]: INFO: Successfully mounted /mnt/state/root-overlay/etc/balena-supervisor.
Sep 06 12:00:04 localhost bindmount[608]: INFO: Bindmounting /etc/docker in /mnt/state/root-overlay/etc/docker ...
Sep 06 12:00:04 localhost bindmount[608]: INFO: Successfully mounted /mnt/state/root-overlay/etc/docker.
Sep 06 12:00:04 localhost bindmount[609]: INFO: Bindmounting /etc/hostname in /mnt/state/root-overlay/etc/hostname ...
Sep 06 12:00:04 localhost bindmount[609]: INFO: Successfully mounted /mnt/state/root-overlay/etc/hostname.
Sep 06 12:00:04 localhost bindmount[610]: INFO: Bindmounting /etc/ssh/hostkeys in /mnt/state/root-overlay/etc/ssh/hostkeys ...
Sep 06 12:00:04 localhost bindmount[610]: INFO: Successfully mounted /mnt/state/root-overlay/etc/ssh/hostkeys.
Sep 06 12:00:04 localhost bindmount[611]: INFO: Bindmounting /etc/udev/rules.d in /mnt/state/root-overlay/etc/udev/rules.d ...
Sep 06 12:00:04 localhost bindmount[611]: INFO: Successfully mounted /mnt/state/root-overlay/etc/udev/rules.d.
Sep 06 12:00:04 localhost bindmount[612]: INFO: Bindmounting /home/root/.rnd in /mnt/state/root-overlay/home/root/.rnd ...
Sep 06 12:00:04 localhost bindmount[612]: INFO: Successfully mounted /mnt/state/root-overlay/home/root/.rnd.
Sep 06 12:00:04 localhost bindmount[613]: INFO: Bindmounting /home/root/.ssh in /mnt/state/root-overlay/home/root/.ssh ...
Sep 06 12:00:04 localhost bindmount[613]: INFO: Successfully mounted /mnt/state/root-overlay/home/root/.ssh.
Sep 06 12:00:04 localhost bindmount[614]: INFO: Bindmounting /var/lib/chrony in /mnt/state/root-overlay/var/lib/chrony ...
Sep 06 12:00:04 localhost bindmount[614]: INFO: Successfully mounted /mnt/state/root-overlay/var/lib/chrony.

Can there be a problem how or when data partition is mounted as it is already mounted when resin-partition-mounter try to mount it?

I finaly solved it temporary by running resize2fs /dev/mmcblk1p6. This expanded the filesystem on my data partition and the device can download the container.

So I guess I have to do some changes in my custom yocto layer to fit the change made in BalenaOS 2.59.0 that start omiting fs check and resize if partition is mounted.

I can override the resin-filesystem-expand - script, but I guess there is a better ways like not mounting the partition before running the script? Any suggestions?

1 Like

hi, is this a Balena supported device type? If not and you are building a custom OS, have you considered contributing this device so that it can be added as a community supported device to balenaCloud (see Device Types and Versioning - Balena Documentation)?
Once the device is a community supported device we will be in a better position to support you.

Hi, it’s not the first time I get this question from your team and maybe it’s time to contributing and add it as a community device.
The problem is that the board we are using is customized for a specialized commercial B2B product.
We are using a System-On-Module but there are development boards with the SOM mounted. Maybe adding one of those development boards could be of interest for the community. Unfortunately our board is not 100% compatible with the development board as we have removed functions to use IOs for other product specific functions so it will not always help us in our development, maybe it just gives us more work.

The SOM that we are using can be found here: https://www.karo-electronics.com/computer-on-modules/qsmp-series

The development boards this SOM can be found here: https://www.karo-electronics.com/computer-on-modules/qsmp-series#tab-content-10922

@alexgg what do you think? and how about my problem above?

1 Like

hi, I think the way forward here would be for your team to contribute the dev board support as a community device, and then to maintain your own layer with product specific customizations.
This would allow us to offer support on the community device, for example when having to adapt it to new balenaOS releases. Otherwise it is difficult for us to provide support as it is effectively a customOS.

1 Like