BalenaOS on Hummingboard Gate/Edge (SolidSense gateway by SolidRun) stuck in update loop

Currently trying to get BalenaOS up and running on SolidRun’s SolidSense IoT gateway. I was hopeful that it would work with BalenaOS out of the box because the underlying hardware is just a Hummingboard Gate/Edge.

When I insert the SD card the resin-image-flasher executable runs, provisions the device, and attempts to flash the image to the internal eMMC on the hummingboard. Then it reboots the device as normal. The first issue is that the Hummingboard Gate/Edge has configurable boot select jumpers that need to be physically changed to boot from eMMC, so if the device boots again from the SD card it attempts to provision itself again and creates a whole new device in the Balena dashboard. When I power off the device, change the jumpers, and power it back on, it fails to boot from eMMC and I’m stuck at the uboot command line. From the UBoot command line I can see that the image was loaded to eMMC, but uboot does not seem to want to load the OS.

I have a workaround for the problem above:

  • Insert SD card with BalenaOS image and power up the board.
  • Allow it to provision itself and when it restarts power off the device.
  • Manually remove the resin-image-flasher executable from the balenaOS image flash-boot folder.
  • Re-insert SD card and power up hummingboard.

This workaround brings the board online as the device it just provisioned itself as. The issue I’m facing now is an infinite update loop when the image is attempting to update but reports that it is running out of space.

06.08.19 11:34:47 (-0600) Downloading image ‘registry2.balena-cloud.com/v2/4f184cf667e4059c5d430a7cf617ab1c@sha256:bec45c512ccd912492d3e4369cb5926ed86f754ab1ee829f09cca325889cd108
06.08.19 11:39:14 (-0600) Failed to download image ‘registry2.balena-cloud.com/v2/4f184cf667e4059c5d430a7cf617ab1c@sha256:bec45c512ccd912492d3e4369cb5926ed86f754ab1ee829f09cca325889cd108’ due to ‘failed to register layer: Error processing tar file(exit status 1): write /usr/lib/jvm/java-8-openjdk-armhf/lib/tools.jar: no space left on device’

I have the image flashed on to a 16gb sd card. Is there something special I need to do to expand the partitions/ which partitions would need to be expanded?

Hello,
If I understand correctly, you boot from the SD card in all cases.
What is the point of provisioning the eMMC then?

One thing you could try is extracting the image that is being flashed by the flasher (it is in /opt/resin-image-solidrun-imx6.resinos-img on the second partition of the flasher image), write it to an SD card and boot from it.
It will hopefully resize its data partition on boot.
I’m not sure the above will work but it may be worth trying.

Booting from eMMC is not completely necessary, but it would add to the physical security of our devices deployed in the field.

Thanks for the suggestion and I will try it out. Will I also need to copy the SPL and UBoot images to the SD card?

Hi @scarson I think to get the SolidSense Gateway working, you would probably need to follow the device contribution guide and add a new machine type to https://github.com/balena-os/balena-fsl-arm . Something similar to https://github.com/balena-os/balena-fsl-arm/pull/70 . Unfortunately the HummingBoard image we provide is for the older HummingBoard i2, so I would imagine it wouldn’t work out of the box or would at the very least need a different device tree overlay for the hardware differences.

@zvin your solution worked to solve the updating problem! The device boots and I can access the host locally and remotely. I have not tried to push our app that uses bluetooth connectivity features, but I fear that is where I’ll have issues due to the device tree. @shaunmulligan That being said I have proper .dtb file for the Hummingboard 2 hardware used on the Hummingboard Edge/Gate. Can I simply copy that into the image alongside the other device trees that are listed?

I’d be more than happy to contribute to the project as well, if that is what is needed.

@scarson I’m not super familiar with how the hummingboard boots, so I can’t say with confidence. One other thing to note is that the hostOS update system will overwrite files in the boot partition and rootFS on updates, so manual alterations of the image will make updates impossible. This is also why I recommend the route of contributing a COMMUNITY board type (more info here: https://www.balena.io/docs/reference/hardware/versioning/#community ) which would then be shown in the dashboard and have OS updates, etc.

@zvin How does the device provision itself with Balena on initial startup? The issue I am having with your solution is that a new device won’t provision itself with Balena and won’t show up on the Dashboard when booting from /opt/resin-image-solidrun-imx6.resinos-img.

I can work around this, too but it is not a sustainable solution.

@shaunmulligan I am currently exploring the ability to contribute the Hummingboard Gate/Edge as a community board type because I understand that this is the most sustainable and scalable solution.

@scarson yup a COMMUNITY board is definitely the most sustainable (we also have private device support, where we do the maintenance, but you need to chat to the sales team about that)

In terms of the device provisioning. The supervisor container should do the provisioning, provided there is a valid config.json in /mnt/boot/ . When the device is booted can you run journalctl -u resin-supervisor --no-pager to get the logs from the supervisor agent and also do a cat /mnt/boot/config.json and paste it here, but remember to hide a keys in that file before posting it here.