I am trying the balena-engine image delta command. I am on a balena device that is in local mode. THe device is on balenaOS 2.29.2+rev3
balena-engine image delta :latest local_image_main:latest
The command completes fingerprinting and then lists all the layers. The command ends with
Untar error on re-exec cmd: fork/exec /proc/self/exe: cannot allocate memory
Hi @vdsilva what device type are you trying that on and what sizes are the images? From the error it seems like there isn’t enough RAM to be able to compute the delta
Device type is a Beaglebone Green.
balena-engine images responds with 4 images
local_image_main Size 643MB
private registry image Size 638MB
balenalib/beaglebone-green-debian-python 638MB (3.6.8-stretch-build)
Supervisor v9.0.1 Size 53.5Mb
The balenalib image is not necessary. My local_image_main is built off my private registry image. My private registry image is a ‘clone’ of the balena lib image on the device.
I could try removing the debian-python image and try the command again.
hmm. I don’t think removing images will help as that only frees up disk space. I think the core problem is shortage of RAM. The BBG only has 512MB and probably about 200MB of that is running the OS services. From what I now about the delta command, it will create a stream of the image in memory, so I guess it could be that it can’t fit the whole 643MB image in active memory. But this is just me guessing at this point. We would have to get the experts on balena-engine ( @petrosagg and @robertgzr ) to give us more insight.
thanks @shaunmulligan. I’ll wait for one of your colleagues to chime in.
Hi @vdsilva. I think the problem is that the engine tries to fork itself into a helper process (the untar process) but because when this happens the parent process has already allocated a fair amount of memory for computing the delta fingerprint, it fails. However this is soft of an artifact from how linux forks processes and CoW memory. I think you can workaround this issue with sysctl vm.overcommit_memory=1 which tells the kernel to never deny memory allocations but I wouldn’t leave this setting on for production devices
@petrosagg Can you give me details on how would I apply this change? I’m just try to play around with balena-engine and will remember to not leave this in on in production.
@vdsilva you should be able to just run that sysctl vm.overcommit_memory=1 as a command in the hostOS on the device.
This is what I get when i run the command as root using the Com port on the Beaglebone Green.
sysctl: error: ‘vm.overcommit_memory=1’ is an unknown key
Hi. We have released version 2.43.0+rev2 for this device. Can you please check the issue on this latest release please? Thank you
Hi, did you have a chance to test on this new version?
Circling back one last time on this. Did you have a chance to check the latest version and see if it has the same problem?
We released version 2.47.1+rev1 for the beaglebone black device type. This release has by default vm.overcommit_memory=1