Resin OS Host Update Not Enough Space Beaglebone Black

A clean upgrade from 2.7.5+rev2 to 2.12.3+rev1 cannot be performed on beaglebone black due to not enough space being available on the device.

root@788b700:/mnt/data/resinhup# cat upgrade-2.x.sh.20180509_145201.log
[000000001][LOG]Loading info from config.json
[000000001][LOG]Loading info from device-type.json
[000000001][LOG]Device type check: OK
[000000001][LOG]Target OS version "2.12.3+rev1" OK.
[000000001][LOG]VARIANT_ID: prod
[000000001][LOG]Host OS version "2.7.5+rev2" OK.
[000000001][LOG]Checking for manifest of registry.hub.docker.com/resin/resinos:2.12.3_rev1-beaglebone-black
[000000003][LOG]Manifest found, good to go...
[000000003][LOG]No resin-device-progress fix is required...
[000000003][LOG]No supervisor updater fix is required...
[000000003][LOG]hostapp-update command exists, use that for update
[000000004][LOG]No device-specific pre-update fix for beaglebone-black
[000000004][LOG]Do not have docker-host running; legacy mode
[000000004][LOG]Clean inactive partition
[000000004][LOG]Balena migration
[000000004][LOG]Stopping supervisor and related services...
2.12.3_rev1-beaglebone-black: Pulling from resin/resinos
cbf5856c503a: Pulling fs layer
cbf5856c503a: Ready to download
cbf5856c503a: Pull complete
Digest: sha256:7816c5a0128a341b5349359eea428c0df91bbc3b18ea010388f25e2b2165e566
Status: Downloaded newer image for registry.hub.docker.com/resin/resinos:2.12.3_rev1-beaglebone-black
/mnt/sysroot/active is not a mountpoint
[000000222][LOG]Starting hostapp-update within a container
time="2018-05-09T14:55:50.879506270Z" level=info msg="libcontainerd: new containerd process, pid: 21"
time="2018-05-09T14:55:52.964986341Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
time="2018-05-09T14:55:53.018783065Z" level=info msg="Loading containers: start."
time="2018-05-09T14:55:54.274736858Z" level=info msg="Loading containers: done."
time="2018-05-09T14:55:54.845440697Z" level=info msg="Docker daemon" commit=73136d1985f28a34705f537e90636cf208734048 graphdriver(s)=aufs version=17.06.0-dev
time="2018-05-09T14:55:54.851482637Z" level=info msg="Daemon has completed initialization"
time="2018-05-09T14:55:54.969681667Z" level=info msg="API listen on /var/run/balena-host.sock"
Starting hostapp-update
Error processing tar file(exit status 1): write /607b01dee0d4e1d7337eccb9e9285b105223ce093f0c3ec46660de1e71e3a337/layer.tar: no space left on device
invalid reference format
[000000434][ERROR]Update based on hostapp-update has failed...
root@788b700:/mnt/data# du -sh *
4.0K    apps.json
753M    docker
16K     lost+found
124K    resin-data
32K     resinhup
root@788b700:/mnt/data# df -h
Filesystem                         Size  Used Avail Use% Mounted on
devtmpfs                           218M     0  218M   0% /dev
/dev/disk/by-partuuid/3a3bb6c2-03  295M  252M   23M  92% /
/dev/disk/by-label/resin-state      19M  199K   17M   2% /mnt/state
tmpfs                              244M  108K  244M   1% /dev/shm
tmpfs                              244M  4.6M  239M   2% /run
tmpfs                              244M     0  244M   0% /sys/fs/cgroup
tmpfs                              244M     0  244M   0% /tmp
tmpfs                              244M   68K  244M   1% /var/volatile
/dev/mmcblk1p1                      40M  506K   40M   2% /mnt/boot
/dev/mmcblk1p2                     295M  2.2M  273M   1% /mnt/sysroot/inactive
/dev/mmcblk1p6                     1.1G  756M  240M  76% /mnt/data

Hi,

Could you share with us the device’s UUID and, as well, activate support access so we can take a look on the device?

Regards,
Andrei

Hi, the issue is that /data (mmcblk1p6) doesn’t have enough space as it is for the 2.7.5->latest update (which requires some migrations, and has to use space on /data during the process, unfortunately, but only temporarily). In this case your device is a very low storage space BBB (2G eMMC, by the look of it?).

It can be fixed up, updated, and in future updates this won’t be an issue. I’ll send you a private message, can send us the UUID of the device, and we’ll update it for you.

Hope this helps!

Thanks so much for your help. I’ve followed up via private message. I’d be helpful for the forum if you’d share the manual steps taken to resolve this issue in case any other users have the same problem.

Is it possible to use beaglebone black with external SD storage instead of eMMC using resin OS? I’m not aware of how to accomplish that, but I’d gladly use SD storage as the boot volume if this was possible.

The device is now fixed up, and should be all good. I think only people running into this will be:

  • very small storage device
  • updating from <2.9.0 to >=2.9.0 version

Neither is very common, and the later should not be the case for everyone who sets up devices now (and start at >=2.9.0)

The manual steps were:

  • start the device up, and have it in “online” state
  • log into the HostOS
  • stop the supervisor and updater:
systemctl stop resin-supervisor
systemctl stop update-resin-supervisor.timer
  • remove the running containers and images
docker rm -f $(docker ps -a -q)
docker rmi -f $(docker images -q)
  • then finally start the update from the dashboard as before, this time it should work.
  • upon reboot, the device will redownload the user application image, and :checkered_flag:

Hope this helps!

Unfortunately at the moment there’s no way to run the BeagleBone from external storage, but we do investigate things…

The device is now fixed up, and should be all good. I think only people running into this will be:

  • very small storage device
  • updating from <2.9.0 to >=2.9.0 version

Neither is very common, and the later should not be the case for everyone who sets up devices now (and start at >=2.9.0)

The manual steps were:

  • start the device up, and have it in “online” state
  • log into the HostOS
  • stop the supervisor and updater:
systemctl stop resin-supervisor
systemctl stop update-resin-supervisor.timer
  • remove the running containers and images
docker rm -f $(docker ps -a -q)
docker rmi -f $(docker images -q)
  • then finally start the update from the dashboard as before, this time it should work.
  • upon reboot, the device will redownload the user application image, and :checkered_flag:

Hope this helps!

Unfortunately at the moment there’s no way to run the BeagleBone from external storage, but we do investigate things…

Thanks for all your help. The device appears to be operating correctly now. It’s good to hear that this issue will no longer occur in future updates. Running off the SD card on BBB would resolve a number of issues I’ve had with the platform, namely that you cannot deploy moderately sized containers due to the amount of space taken up by the host OS. But I understand that this is a peculiar device which is not top priority.

One thing you can try, if you are deploying images that are relatively large (i.e. larger than ~50% of the data partition, that is ~1.1GB in your case), to use the delete-then-download update strategy. That has a longer update cycle if you deploy an application, and more network traffic, but then you can use pretty much all the data partition (minus the supervisor, that is ~50MB at the moment) and your user data.

Just some hints :slight_smile: let us know if you try it out!

That’s a great suggestion and not something i’ve considered yet because my current workload does function under download-then-kill (the default strategy) but I will keep that in mind in case I run into this issue again.

1 Like

@imrehg My device is currently stuck updating at 50%. I was under the impression that updates would work going forward after what you did but it seems that there are still problems with this device. I have granted support access for 3 hours. The device id is 788b700ae09e32b53a3160a413bd144d.

checking it out

@esetnik the device is not stuck, but it’s updating now, it’s just slow, I see that the image download just close to finish now. Keeping an eye on the device.

Ok it’d be helpful if there was some more feedback on the updater about progress. I’ve been waiting quite a while with it at 50% so it’s hard to tell what’s going on.

There’s not much feedback that can be given as the download progress is pretty opaque (docker pulls information is not that easy to get out of docker). The 50% is “logical stage”, not byte-wise progress.

Also, it seems like the whole update from start to finish (the device is rebooting now) took 12 minutes, that’s pretty fast usually for an update. Not saying it couldn’t be faster, but for the storage and syncing it does for safety it is on par with my experience of updates.

Thanks, I’ll keep that in mind re. the expected performance of updates.

The BeagleBone is not known for its fast storage, unfortunately.

But I definitely agree with you that more detailed progress would be useful. In my case I usually log in the host OS, and do a watch -5 df /mnt/sysroot/inactive to see how the pull is progressing on that partition. This is a sort of behind the scenes trick, but anyways, might be helpful. Or hopefully in the future things can be faster too. I know that balenaEngine includes some improvements, that are fast AND safe (this version you were on mostly goes for safe).