Manual update of custom host os fails on balena load

Product support balenaOS
1

I’m in the process of upgrading my custom host os to a newer version using hostapp-update, but it fails with error:

Error processing tar file(exit status 1): write /lib/modules/4.19.71/kernel/drivers/gpu/drm/vc4/vc4.ko: no space left on device

It seems to fail when trying to balena load the image and I can reproduce the issue on the command line:

root@hltest-mc-2:~# DOCKER_HOST="unix:///var/run/balena-host.sock" balena load -i /resin-data/data/helioslite-raspberrypi-2.46.1+rev5~hl1-dockerimg.tar 
f1d9e8a0f1fd: Loading layer [===================>                               ]  85.23MB/223MB
Error processing tar file(exit status 1): write /lib/modules/4.19.71/kernel/drivers/gpu/drm/vc4/vc4.ko: no space left on device

This is weird because I have plenty of space on the /mnt/sysroot/inactive partition:

root@hltest-mc-2:~# df -h /mnt/sysroot/inactive
Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p3  300M  2.2M  278M   1% /mnt/sysroot/inactive

The image I’m trying to update is not that big so I think this should fit in the /mnt/sysroot/inactive partition:

root@hltest-mc-2:~# du -h /resin-data/data/helioslite-raspberrypi-2.46.1+rev5~hl1-dockerimg.tar 
210M	/resin-data/data/helioslite-raspberrypi-2.46.1+rev5~hl1-dockerimg.tar

I’ve noticed this similar issue on the forum but in my case the docker storage has already been cleaned up:

root@hltest-mc-2:~# DOCKER_HOST="unix:///var/run/balena-host.sock" balena system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              0                   0                   0B                  0B
Containers          0                   0                   0B                  0B
Local Volumes       0                   0                   0B                  0B

My custom host os image is very similar to the official one (just a few services have been added, nothing taking up lot of space). One point to notice though, is that I’m using overlay2 docker storage instead of the default aufs one. Could it explain this upgrade issue?

I’ve tried to watch how the balena load is progressing on /mnt/sysroot/inactive partition with command watch -n 5 df -h /mnt/sysroot/inactive and I noticed that the used space of that partition seemed to increase quicker than the balena load was progressing: even before balena load progress bar was displayed, the “used space” had already increased from 2.2M to 216M:

Every 5s: df -h /mnt/sysroot/inactive                                                                                                                                                       2020-01-29 15:02:54

Filesystem      Size  Used Avail Use% Mounted on
/dev/mmcblk0p3  300M  216M   64M  78% /mnt/sysroot/inactive

Then the balena load progress bar is displayed and /mnt/sysroot/inactive used space continues to increase until 100%…

So it’s like if the docker host os image was loaded twice…

Any idea?

2

Also note that the existing host os version is based on 2.13.6:

root@hltest-mc-2:~# cat /mnt/boot/os-release 
ID="resin-os"
NAME="Resin OS"
VERSION="2.13.6+rev1~hl1"
VERSION_ID="2.13.6"
PRETTY_NAME="Resin OS 2.13.6+rev1~hl1"
RESIN_BOARD_REV="6ac8ecd"
META_RESIN_REV="e167839b"
SLUG="raspberry-pi"
MACHINE="raspberrypi"
VARIANT="Production"
VARIANT_ID="prod
5

I’ve continued to investigate on the disk space used before balena load progress bar was displayed, so I’ve generated the list of files in the /mnt/sysroot/inactive partition before and while running the balena load command. Here is the result:

root@hltest-mc-2:~# find /mnt/sysroot/inactive > /tmp/list.1
root@hltest-mc-2:~# # starting balena load in another shell
root@hltest-mc-2:~# find /mnt/sysroot/inactive > /tmp/list.2
root@hltest-mc-2:~# diff /tmp/list.1 /tmp/list.2
--- /tmp/list.1
+++ /tmp/list.2
@@ -36,6 +36,12 @@
 /mnt/sysroot/inactive/balena/image/overlay2/imagedb/content/sha256
 /mnt/sysroot/inactive/balena/image/overlay2/distribution
 /mnt/sysroot/inactive/balena/tmp
+/mnt/sysroot/inactive/balena/tmp/docker-import-267812713
+/mnt/sysroot/inactive/balena/tmp/docker-import-267812713/813185192bf92f46417397d771b5697b20c24f8557378f31b5c432f699fdb296.json
+/mnt/sysroot/inactive/balena/tmp/docker-import-267812713/fd8004b4155704476346c4ad70d0eb4a65dcd7465e548731dfe9c1c24f581f80
+/mnt/sysroot/inactive/balena/tmp/docker-import-267812713/fd8004b4155704476346c4ad70d0eb4a65dcd7465e548731dfe9c1c24f581f80/VERSION
+/mnt/sysroot/inactive/balena/tmp/docker-import-267812713/fd8004b4155704476346c4ad70d0eb4a65dcd7465e548731dfe9c1c24f581f80/json
+/mnt/sysroot/inactive/balena/tmp/docker-import-267812713/fd8004b4155704476346c4ad70d0eb4a65dcd7465e548731dfe9c1c24f581f80/layer.tar
 /mnt/sysroot/inactive/balena/network
 /mnt/sysroot/inactive/balena/network/files
 /mnt/sysroot/inactive/balena/network/files/local-kv.db

So it seems balena load will first untar the docker image before actually loading it, so this explains why it requires much higher size available than the final size of the image.
A possible workaround would be to used the different location for the /mnt/sysroot/inactive/balena/tmp dir, which would not sit in the /mnt/sysroot/inactive partition: maybe a symlink would do the job.

6

Hi @bbinet, you are on good track to a solution here : )

Sounds like the host update is interrupted and there has been leftover data on the inactive partition. Then in the next update attempt, the update is running out of space. You could do a manual cleanup of that partition you point at via:

systemctl stop balena-host
rm -rf /mnt/sysroot/inactive/balena
systemctl start balena-host

Let us know how it goes and if we could help out further.

8

@gelbal No, there is no leftover data on the inactive partition.

I’ve temporarily implemented the symlink workaround which works well:

root@hltest-mc-2:~# rm -fr /mnt/sysroot/inactive/balena/tmp
root@hltest-mc-2:~# ln -s /mnt/data/docker/tmp /mnt/sysroot/inactive/balena/tmp
root@hltest-mc-2:~# hostapp-update -f /resin-data/data/helioslite-raspberrypi-2.46.1+rev5~hl1-dockerimg.tar
[INFO] Checking if boot partition can accommodate the new update... success.
[INFO] Deploying /mnt/boot/start.elf... done.
[INFO] Deploying /mnt/boot/cmdline.txt... done.
[INFO] Deploying /mnt/boot/kernel.img... done.
[INFO] Deploying /mnt/boot/bootcode.bin... done.
[INFO] Deploying /mnt/boot/bcm2708-rpi-zero-w.dtb... new file... done.
[INFO] Deploying /mnt/boot/config.txt...file blacklisted. Ignoring.
[INFO] Deploying /mnt/boot/image-version-info... done.
[INFO] Deploying /mnt/boot/bcm2710-rpi-3-b-plus.dtb... done.
[INFO] Deploying /mnt/boot/config.json...file blacklisted. Ignoring.
[INFO] Deploying /mnt/boot/start_cd.elf... done.
[INFO] Deploying /mnt/boot/bcm2835-bootfiles-20190828.stamp... new file... done.
[INFO] Deploying /mnt/boot/splash/resin-logo.png...file blacklisted. Ignoring.
[INFO] Deploying /mnt/boot/resin-image... done.
[INFO] Deploying /mnt/boot/bcm2710-rpi-cm3.dtb... done.
[INFO] Deploying /mnt/boot/start_x.elf... done.
[INFO] Deploying /mnt/boot/bcm2711-rpi-4-b.dtb... new file... done.
[INFO] Deploying /mnt/boot/os-release... done.
[INFO] Deploying /mnt/boot/bcm2708-rpi-b-plus.dtb... done.
[INFO] Deploying /mnt/boot/fixup_x.dat... done.
[INFO] Deploying /mnt/boot/fixup_cd.dat... done.
[INFO] Deploying /mnt/boot/device-type.json... done.
[INFO] Deploying /mnt/boot/bcm2709-rpi-2-b.dtb... done.
[INFO] Deploying /mnt/boot/resinos.fingerprint... overwriting modified file... done.
[INFO] Deploying /mnt/boot/bcm2708-rpi-b.dtb... overwriting modified file... done.
[INFO] Deploying /mnt/boot/boot.scr... new file... done.
[INFO] Deploying /mnt/boot/overlays/pitft22.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/dht11.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi0-cs.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/sdio.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/ads1015.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi1-2cs.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi1-1cs.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/smi-nand.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/pps-gpio.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/allo-digione.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/allo-boss-dac-pcm512x-audio.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/audremap.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/piscreen2r.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/rpi-cirrus-wm5102.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/mz61581.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/dpi24.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/dpi18.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi4-2cs.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/i2c-bcm2708.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/hyperpixel4-square-pi4.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/rpi-poe.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/justboom-digi.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/dwc-otg.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/rpi-dac.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/vc4-kms-v3d.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/pi3-miniuart-bt.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/allo-piano-dac-plus-pcm512x-audio.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/sdhost.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/mcp2515-can0.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi0-hw-cs.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/pitft35-resistive.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi3-1cs.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/mcp23017.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/i2c-rtc-gpio.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi5-2cs.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/smi.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/ads1115.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/audioinjector-addons.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/ads7846.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi-gpio35-39.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/rpi-sense.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/adau1977-adc.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/gpio-poweroff.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/rpi-proto.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/pi3-act-led.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/i2c-rtc.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/pisound.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/gpio-key.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/i2c-sensor.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/hyperpixel4-pi3.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/smi-dev.dtbo... done.
[INFO] Deploying /mnt/boot/overlays/mcp3008.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/enc28j60.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/i2c0-bcm2708.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/tpm-slb9670.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/papirus.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/hy28a.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/dionaudio-loco.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/midi-uart1.dtbo... done.
[INFO] Deploying /mnt/boot/overlays/uart0.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/qca7000.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/pwm-2chan.dtbo... done.
[INFO] Deploying /mnt/boot/overlays/rpi-backlight.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/pi3-disable-bt.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/hyperpixel4-pi4.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/hifiberry-digi.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi2-2cs.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/mcp2515-can1.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/w1-gpio.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/iqaudio-dac.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/hifiberry-amp.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/i2c1-bcm2708.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/rotary-encoder.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/dionaudio-loco-v2.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/vga666.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/hifiberry-dacplus.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/hy28b.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/w1-gpio-pullup.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/adau7002-simple.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/rpi-display.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/allo-piano-dac-pcm512x-audio.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi-rtc.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/piscreen.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/uart1.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/mcp23s17.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/i2s-gpio28-31.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/dwc2.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/justboom-dac.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/i2c-gpio.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/rpi-tv.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/hifiberry-dac.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/bmp085_i2c-sensor.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/sc16is750-i2c.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/wittypi.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/pitft28-resistive.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/pi3-disable-wifi.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/pitft28-capacitive.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/iqaudio-digi-wm8804-audio.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/goodix.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi6-1cs.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/spi5-1cs.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/tinylcd35.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/sc16is752-spi1.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/akkordion-iqdacplus.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/i2c-pwm-pca9685a.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/mmc.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/fe-pi-audio.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/iqaudio-dacplus.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi3-2cs.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/spi2-3cs.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/sdtweak.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/gpio-ir.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/googlevoicehat-soundcard.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/mpu6050.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi1-3cs.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/hifiberry-digi-pro.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/spi4-1cs.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/midi-uart0.dtbo... done.
[INFO] Deploying /mnt/boot/overlays/vc4-fkms-v3d.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/at86rf233.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/enc28j60-spi2.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/pwm.dtbo... done.
[INFO] Deploying /mnt/boot/overlays/spi2-1cs.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/hyperpixel4-square-pi3.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/spi6-2cs.dtbo... new file... done.
[INFO] Deploying /mnt/boot/overlays/rra-digidac1-wm8741-audio.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/audioinjector-wm8731-audio.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/gpio-shutdown.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/i2c-mux.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/overlays/rpi-ft5406.dtbo... overwriting modified file... done.
[INFO] Deploying /mnt/boot/bcm2710-rpi-3-b.dtb... overwriting modified file... done.
[INFO] Deploying /mnt/boot/system-connections/resin-sample.ignore... done.
[INFO] Deploying /mnt/boot/system-connections/README.ignore... done.
[INFO] Deploying /mnt/boot/fixup.dat... overwriting modified file... done.
[INFO] Deploying /mnt/boot/bcm2708-rpi-cm.dtb... overwriting modified file... done.
[INFO] sshd_migrate_keys: Converting dropbear host key...
Key is a ssh-rsa key
Wrote key to '/mnt/state/root-overlay/etc/ssh/hostkeys/ssh_host_rsa_key'
[INFO] sshd_migrate_keys: Migrated dropbear host RSA key to openssh.
[INFO] Running rollback hook 
[INFO] Switching uboot root partition index to 3...... done.
New hooks ran successfully.

But this is a manual workaround and I’m interested to understand why the balena team don’t hit this issue when updating host os?
Are you doing anything special with the balena tmp directory prior to running hostapp-update?

9

I guess you don’t face this issue because you are not doing offline updates, but online updates with hostapp-update -i:

“hostapp-update -i” probably don’t need to untar any data /mnt/sysroot/inactive/balena/tmp so this works for you.

Should I create an issue in meta-balena repository to fix “hostapp-update -f”?
I can also submit a pull request with my symlink workaround if you think this would be an acceptable solution?

10

Hi, @bbinet thanks for the thorough investigation on this. Yes please, can you create an issue in the meta-balena repo summarizing this issue and linking here. We would welcome any PRs but if you wanted to discuss this fix further before proceeding with the PR then the GitHub issue would be the best way to do it and would be seen by the relevant engineers.

12

Here we go: https://github.com/balena-os/meta-balena/pull/1826

1 Like
13

hey @bbinet,
thanks so much for your contribution! The maintainers of the meta-balena repo will review the PR and get back to you.

Thanks and regards
Rahul