Running out of memory during build via balena local push

Product support
,
1

I’m running out of memory while performing a build on a device in local mode via:

balena push <device-ip>

The build hangs indefinitely when it runs out of memory.

I’m wonder if there is a way to use a swapfile during a local build (or otherwise increase the memory limit).

Cheers,
Will

Related thread: Is it possible to setup swap on resin.io / ResinOS?

5

Hi,
Would it possible to get the logs up until the build hangs? It would be useful to check which components actually hangs.
Kind regards,
Theodor

6

Hi @telphan – thanks for the quick reply :slight_smile:

Would you mind pointing me at the location of the log file you’re looking for?

7

I just mean the output of the balena push command.

9

It’s not a particularly informative log, but this is the output.

$ balena push 192.168.1.125
[Info]    Starting build on device 192.168.1.125
[Info]    Creating default composition with source: .
[Build]   [main] Step 1/6 : FROM wlisac/raspberrypi3-64-swift:5.0
[Build]   [main]  ---> 56c89f817152
[Build]   [main] Step 2/6 : WORKDIR /package
[Build]   [main] Removing intermediate container a27b9736856f
[Build]   [main]  ---> 19010852ce84
[Build]   [main] Step 3/6 : COPY . ./
[Build]   [main]  ---> 88e31d49e608
[Build]   [main] Step 4/6 : RUN swift build
[Build]   [main]  ---> Running in 0a4bab3821aa
[Build]   [main] [1/2] Compiling Swift Module 'Hello'

The build hangs indefinitely – I believe due to running out of memory.

If I build the same Dockerfile using Docker directly on a Raspberry Pi 3 running a 64 bit host OS (Ubuntu 18.04) with a swapfile enabled, the build is able to complete.

As an aside – there is a known issue with Swift 5.0 using excessive memory on aarch64 Raspberry Pi, so this problem may eventually solve itself, but it would still be great to know if it’s possible to increase the memory limit during local builds.

Thanks for the help,
Will

15

Hi @wlisac, could connect to the device locally, and see whether it is really running out of memory, or anything else is going on.

I’ve put together a small “hello” project based on what you posted above

And this builds totally fine with balena push IP, so not everything is running out of memory, at least it seems. I wonder if you want to share your project so we can try it locally as well.

Also, what version of the OS and the CLI are you using?

Swap

Using swap is an interesting idea. Currently you would have to do it manually. Probably it would work inside the running device, but the easiest is (since it’s a local device for you), if you take the SD card out, and resize the partitions / create the new ones in your development machine. For example using gparted, shrink the resin-data partition, and create a new logical partition of the linux-swap kind in the freed up space afterwards.

When you start up the device, you’ll have to set swapon /dev/mmcblk0p7 (if you did as mentioned above, it will be the 7th partition), and that enables swap, you can see it with swapon -s and cat /proc/meminfo | grep Swap that it’s in use. Unfortunately on every boot you will have to set this swapon ... with this device, as it’s not by default using it.

For local development devices it should be okay, I presume? Or what do you think?

1 Like
18

Wow – thanks for such a great reply and for even setting up your own Swift sample project. Impressed!

You’re right that not all builds run out of memory – I’ve opened up a PR on your sample project that adds a few more source files. That should be enough to cause the Pi to run out of memory during the build. :sweat_smile:

I’ll have to try the manual swapfile approach and report back.

Folks are working on this excessive memory consumption issue (for example, in this PR to SPM), so I’m hopeful that the swapfile solution will just be temporary.

Here is the version info you requested:

OS version: balenaOS 2.31.5+rev5
Balena CLI version: 10.0.0

Thanks again for the help.

– Will