Cannot build big images in local mode

Project help
1

I’ve got a Dockerfile that uses node and installs and compiles several modules with NPM.
The problem is it takes a very long time (at least 10 minutes) on my raspberry pi 3b+ and the cli loses connection while it’s happening, and the image never gets built.

I can use balena cloud to build the image but that takes a good 4 minutes between finishing writing code and seeing it running. The live code push in local mode is a really nice experience that I’d like to work.

The reason the image build fails is because systemd kills the balena engine before it completes.
There’s a related issue here https://github.com/balena-os/balena-engine/issues/196

In the journalctl logs you can see:
balena.service: Watchdog timeout (limit 6min)

From some investigation I can see that (i think) systemctl is killing it because the health check that healthdog is supplying it isn’t happening (I guess /usr/lib/balena/balena-healthcheck is very very slow or the engine isn’t replying?)

From systemctl show balena.service:
WatchdogUSec=6min

I thought I’d change the systemd settings for the balena service and increase the timeout. It’s defined in /etc/systemd/system/balena.service.d/balena.conf but that file’s mounted as read only, which is fair enough … but I can’t figure out how to edit it because I don’t know where it actually exists. I thought what’s in /etc would either be from /mnt/sysroot/active/etc or /mnt/state/root-overlay/etc/ but it’s not in either place.

4

Hello @tboothman, are you using a fast SD card? If not, perhaps upgrading to a higher quality SD card may help speed up the build to avoid hitting the timeout. I’m not sure the timeout can be easily increased, but I’ll ask internally (including your question about the balena.conf location) and get back to you.

7

I ended up with this:

mount -o remount,rw /
echo ‘WatchdogSec=100min’ >> /etc/systemd/system/balena.service.d/balena.conf
Then restart the device.

It took about an hour to build the layer in the end … but at least it finished.

The SD card is a Kingston industrial and supposedly has ‘90MB/s read, 45MB/s write’. I don’t know if you’d consider that slow, it seems in line with regular SD cards.

8

I’m glad you were able to get it to build. We recommend using Sandisk Extreme Pro SD cards, which claim to have a write speed up to 90MB/s. It may be worth giving one a try to see if it improves your speed.

10

Sorry for not responding for long. It was as you suggested - a slow SD card. I got a Sandisk extreme (although other team members have ultra plus and that’s fine too) and it’s all good now. Still takes 15 minutes to build the npm modules but the Balena cloud servers take about half that so I’m fine with it.

Thanks for your help

12

Happy to hear it worked @tboothman!
Georgia