Notably increment of RAM usage in newest Resin OS versions

Hi,

I am analysing the memory usage of my devices, and I have noticed an increment of memory usage just updating the resin OS version. The difference that I am talking about is around a 10% in a RPI3 (1GB or RAM), comparing:

Oldest image: VERSION HOST OS VERSION Resin OS 2.12.7+rev1; SUPERVISOR VERSION 7.4.3
Newest image: VERSION HOST OS VERSION Resin OS 2.14.3+rev; SUPERVISOR VERSION 7.16.6

Is this a normal or expected behaviour? Might it be configured through some fleet env vars or something? My problem is that I am using around 90% or RAM and sometimes it throws an OOM signal which reboot my devices.

Thank you in advance!

Hi, where do you see the larger memory usage, your application containers use more memory, or the supervisor, or something else?
Any info would be appreciate that we can use to try to reproduce the issue.

We have some example projects that we run to monitor device resource use, and would like to replicate what you see.

The main big change between those versions is that the newer resinOS version is running a newer balena version.

The application containers use exactly the same memory usage percentage (the image for those applications are the same in both devices). I am using top to check that and I am not sure about the rest. I attach the output of top command in oldest and newest version, ordering processes by memory usage (the rest are basically the application containers which use exactly the same amount of RAM):

Oldest:

 PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  767     1 root     S    1254m 138%   0% /usr/bin/balenad --experimental --log-driver=journald -s aufs -H fd:// --dns 10.114.102.1 --bip 10.114.101.1/24 --fixed-cidr=10.114.101.0/25
  783   767 root     S    1002m 111%   0% balena-containerd -l unix:///var/run/balena/libcontainerd/balena-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/balena/libcontainerd/containerd
  700   662 root     S     914m 101%   0% balena-containerd -l unix:///var/run/balena-host/libcontainerd/balena-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/balena-host/libcontainerd/
  662     1 root     S     898m  99%   0% /usr/bin/balenad --delta-data-root=/mnt/sysroot/active/balena --delta-storage-driver=aufs --log-driver=journald -s aufs --data-root=/mnt/sysroot/inactive/balena -H unix:///v
 2785   783 root     S     873m  96%   0% balena-containerd-shim 5c23ebd688e92f880baa580b661b69fc1633f83a139f2303b172b82c011b8f84 /var/run/balena/libcontainerd/5c23ebd688e92f880baa580b661b69fc1633f83a139f2303b172b82
21999   783 root     S     873m  96%   0% balena-containerd-shim a9085da0250041623fe976151b0d1b2eed4d5539e8cf1bf453f54f7f307b96bf /var/run/balena/libcontainerd/a9085da0250041623fe976151b0d1b2eed4d5539e8cf1bf453f54f7
21291   783 root     S     865m  96%   0% balena-containerd-shim a4f5331324e17309a2d8d05b9517a726ac6ae5f4196054620e1f40683d168417 /var/run/balena/libcontainerd/a4f5331324e17309a2d8d05b9517a726ac6ae5f4196054620e1f406
21300   783 root     S     865m  96%   0% balena-containerd-shim 9d74393d9437270f39907a6a52bc9533ff013f80e5af35eb10d6a5553e3b564e /var/run/balena/libcontainerd/9d74393d9437270f39907a6a52bc9533ff013f80e5af35eb10d6a55
22412   783 root     S     865m  96%   0% balena-containerd-shim a6c02c703dfc20f038e3ccf17628457799c40f6145c532edf372bde02c3ce686 /var/run/balena/libcontainerd/a6c02c703dfc20f038e3ccf17628457799c40f6145c532edf372bde
22153   783 root     S     865m  96%   0% balena-containerd-shim 8ffa6a15927ddaa6ccce6c57be3e08cfe7fc8ac9cd4a71ee872e1fdbe091b3df /var/run/balena/libcontainerd/8ffa6a15927ddaa6ccce6c57be3e08cfe7fc8ac9cd4a71ee872e1fd
21137   783 root     S     865m  96%   0% balena-containerd-shim 8b4632fc98af9fc43a9b4906cc549c86f4be1b75dd74600beb735ad4bcc042a2 /var/run/balena/libcontainerd/8b4632fc98af9fc43a9b4906cc549c86f4be1b75dd74600beb735ad
 2776  2732 root     S     865m  96%   0% balena start --attach resin_supervisor
21690   783 root     S     865m  96%   0% balena-containerd-shim e6bc9fdacf17bab67e60fecdcce2e356a68c68fc896ae67d6d314b138baa72fb /var/run/balena/libcontainerd/e6bc9fdacf17bab67e60fecdcce2e356a68c68fc896ae67d6d314b1
21427   783 root     S     865m  96%   0% balena-containerd-shim 0fa1da58d1cc4fc9bc398a41cd4ca561bb3a25e2323ba765399e08a1e1de42f1 /var/run/balena/libcontainerd/0fa1da58d1cc4fc9bc398a41cd4ca561bb3a25e2323ba765399e08a
22257   783 root     S     865m  96%   0% balena-containerd-shim f1f4f298840269bd03b36a7f04b0bcdf1bb69fa91d81f0ffd5f09abcf5d13863 /var/run/balena/libcontainerd/f1f4f298840269bd03b36a7f04b0bcdf1bb69fa91d81f0ffd5f09ab
21035   783 root     S     865m  96%   0% balena-containerd-shim 25f61cfab19b9f971d6fdb3f3b1019bb2c8926e70f843f58b3fedec0f6f7b139 /var/run/balena/libcontainerd/25f61cfab19b9f971d6fdb3f3b1019bb2c8926e70f843f58b3fedec
21282   767 root     S     857m  95%   0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 7500 -container-ip 172.18.0.4 -container-port 7500
21311   767 root     S     857m  95%   0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.18.0.5 -container-port 80
2802  2785 root     S     137m  15%   0% node /usr/src/app/dist/app.js

Newest:

  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  849     1 root     S    1337m 148%   0% /usr/bin/balenad --experimental --log-driver=journald -s aufs -H fd:// --dns 10.114.102.1 --bip 10.114.101.1/24 --fixed-cidr=10.114.101.0/25
  863   849 root     S     906m 100%   0% balena-containerd --config /var/run/balena/containerd/containerd.toml
  726   688 root     S     905m 100%   0% balena-containerd --config /var/run/balena-host/containerd/containerd.toml
  688     1 root     S     897m  99%   0% /usr/bin/balenad --delta-data-root=/mnt/sysroot/active/balena --delta-storage-driver=aufs --log-driver=journald -s aufs --data-root=/mnt/sysroot/inactive/balena -H unix:///v
 2554   863 root     S     873m  96%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/9dc3ffe9f817418080537f9f818ab80495194974d40d6ccf09371af
 3366  3340 root     S     873m  96%   0% balena start --attach resin_supervisor
 3377   863 root     S     865m  95%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/c6c70e7fa6f82f192b101b56d04774af73fbebb1e37e8e5467f3775
 2921   863 root     S     865m  95%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/5cd5b3cea2ee2f480983168a8133c6b88e105227d88704a1707c10d
 2339   863 root     S     865m  95%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/09969f35b064680896f946bcd8d9817f7637e5154080896c65a72f3
 3004   863 root     S     864m  95%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/e28ca063c35280acdbb4f44f40c8917d827978a9fd2664c8119c4bb
 2560   863 root     S     864m  95%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/2457b45f1494f011d0c58562614d6e3ab7f59e3c2ac4a95f09be279
 2036   849 root     S     864m  95%   0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 7500 -container-ip 172.18.0.6 -container-port 7500
 2500   863 root     S     856m  95%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/0db9a2b926e7336edb57965dbdb83e31bc9601acd94239deb6f486b
 2102   863 root     S     855m  94%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/1cb5a5eb6061994fe3137d48a652339cd17181304e6984fce5b31ea
 2421   863 root     S     855m  94%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/0732265d6523659e547fb490f16a9526835fdc58b3746eeea59f066
 3393  3377 root     S     136m  15%   0% node /usr/src/app/dist/app.js

Just to give more information, I have not tried to swap SD cards between these devices. I can try that in order to discard different hardware, or just hardware problems.

@jcozar, the first thing that comes to mind is that in resinOS 2.14.3 balena was upgraded to v17.12 which is obviously based on docker 17.12. My guess is that new changes in balena/docker probably increased the RAM usage, but perhaps @petrosagg can speak a bit more to if this could be the case

I swapped the SD cards between both devices and it happens the same with the OS versions, so there is not a hardware problem.

I have configured both devices exactly in the same way. Again, I report you the output of the top command for the oldest and the newest OS versions:

Oldest:

Mem: 728704K used, 196484K free, 12944K shrd, 42672K buff, 252804K cached
CPU:   4% usr   1% sys   0% nic  94% idle   0% io   0% irq   0% sirq
Load average: 0.07 0.11 0.15 2/402 23726
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  748     1 root     S    1186m 131%   0% /usr/bin/balenad --experimental --log-driver=journald -s aufs -H fd:// --dns 10.114.102.1 --bip 10.114.101.1/24 --fixed-cidr=10.114.101.0/25
  779   748 root     S     986m 109%   0% balena-containerd -l unix:///var/run/balena/libcontainerd/balena-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/balena/libcontainerd/containerd
  672     1 root     S     898m  99%   0% /usr/bin/balenad --delta-data-root=/mnt/sysroot/active/balena --delta-storage-driver=aufs --log-driver=journald -s aufs --data-root=/mnt/sysroot/inactive/balena -H unix:///v
  710   672 root     S     898m  99%   0% balena-containerd -l unix:///var/run/balena-host/libcontainerd/balena-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/balena-host/libcontainerd/
 2471   779 root     S     873m  96%   0% balena-containerd-shim 5c23ebd688e92f880baa580b661b69fc1633f83a139f2303b172b82c011b8f84 /var/run/balena/libcontainerd/5c23ebd688e92f880baa580b661b69fc1633f83a139f2303b172b82
 3474   779 root     S     873m  96%   0% balena-containerd-shim cffd937de4d2b3f0529085bfcee564b4460cd240992cf9e716096b5841abf091 /var/run/balena/libcontainerd/cffd937de4d2b3f0529085bfcee564b4460cd240992cf9e716096b5
 3407   779 root     S     873m  96%   0% balena-containerd-shim 059770f4fc5087964da8444eecf4f6817afb6feb7ccc4d81a6f6e974bfb1fc92 /var/run/balena/libcontainerd/059770f4fc5087964da8444eecf4f6817afb6feb7ccc4d81a6f6e97
 3681   779 root     S     873m  96%   0% balena-containerd-shim 6f31ce31077e15e6a758581971e33eb39a568dda8033c492e739b11a49645efa /var/run/balena/libcontainerd/6f31ce31077e15e6a758581971e33eb39a568dda8033c492e739b11
 3470   779 root     S     873m  96%   0% balena-containerd-shim 394b48afbf50ec9bde961b49ea07ec3ba8338cdda73caa53063129c982cf85a0 /var/run/balena/libcontainerd/394b48afbf50ec9bde961b49ea07ec3ba8338cdda73caa53063129c
 3134   779 root     S     866m  96%   0% balena-containerd-shim c6dc92911963fffe619bbc5f848895cec15bb7efa1d67d2d64d579794f5b77de /var/run/balena/libcontainerd/c6dc92911963fffe619bbc5f848895cec15bb7efa1d67d2d64d5797
 3290   779 root     S     865m  96%   0% balena-containerd-shim a2614ecaba19bcf885ac804ace0dd13ba063bbe55f0e534128490c862a0bafc6 /var/run/balena/libcontainerd/a2614ecaba19bcf885ac804ace0dd13ba063bbe55f0e534128490c8
 3476   779 root     S     865m  96%   0% balena-containerd-shim c0c9fae9c5c1893c7a465532f9c111a5501317885d537906ddef6171f60454fd /var/run/balena/libcontainerd/c0c9fae9c5c1893c7a465532f9c111a5501317885d537906ddef617
 2462  2434 root     S     865m  96%   0% balena start --attach resin_supervisor
 3767   779 root     S     865m  96%   0% balena-containerd-shim 5b300662c2481af2ff34f7a61afbdbf46de49b9a9074ef6c783920d954bbe207 /var/run/balena/libcontainerd/5b300662c2481af2ff34f7a61afbdbf46de49b9a9074ef6c783920d
 3786   779 root     S     865m  95%   0% balena-containerd-shim 122a9fb280e8e3addd5135c463d5da77a0ff32188cf3fcf97200b6dc82c7e085 /var/run/balena/libcontainerd/122a9fb280e8e3addd5135c463d5da77a0ff32188cf3fcf97200b6d
 3367   748 root     S     857m  95%   0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.19.0.4 -container-port 80
 3275   748 root     S     857m  95%   0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 7500 -container-ip 172.19.0.3 -container-port 7500

Newest:

Mem: 825696K used, 98328K free, 7280K shrd, 11336K buff, 64500K cached
CPU:   3% usr   0% sys   0% nic  95% idle   0% io   0% irq   0% sirq
Load average: 0.19 0.19 0.18 2/338 24528
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
  856     1 root     S    1139m 126%   0% /usr/bin/balenad --experimental --log-driver=journald -s aufs -H fd:// --dns 10.114.102.1 --bip 10.114.101.1/24 --fixed-cidr=10.114.101.0/25
  869   856 root     S     906m 100%   0% balena-containerd --config /var/run/balena/containerd/containerd.toml
  688     1 root     S     905m 100%   0% /usr/bin/balenad --delta-data-root=/mnt/sysroot/active/balena --delta-storage-driver=aufs --log-driver=journald -s aufs --data-root=/mnt/sysroot/inactive/balena -H unix:///v
  728   688 root     S     905m 100%   0% balena-containerd --config /var/run/balena-host/containerd/containerd.toml
 2246   869 root     S     873m  96%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/c6c70e7fa6f82f192b101b56d04774af73fbebb1e37e8e5467f3775
 2234  2203 root     S     873m  96%   0% balena start --attach resin_supervisor
 1469   869 root     S     865m  95%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/5cd5b3cea2ee2f480983168a8133c6b88e105227d88704a1707c10d
 1260   869 root     S     865m  95%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/09969f35b064680896f946bcd8d9817f7637e5154080896c65a72f3
 1285   869 root     S     865m  95%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/1cb5a5eb6061994fe3137d48a652339cd17181304e6984fce5b31ea
 1784   869 root     S     865m  95%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/ba4307233508cf87e65a722da31e036fd731e9f9898609005347ffe
 1639   856 root     S     865m  95%   0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.18.0.9 -container-port 80
 1593   869 root     S     856m  95%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/2457b45f1494f011d0c58562614d6e3ab7f59e3c2ac4a95f09be279
 1570   869 root     S     856m  94%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/0732265d6523659e547fb490f16a9526835fdc58b3746eeea59f066
 1438   869 root     S     856m  94%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/9dc3ffe9f817418080537f9f818ab80495194974d40d6ccf09371af
 1337   869 root     S     856m  94%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/0db9a2b926e7336edb57965dbdb83e31bc9601acd94239deb6f486b
 1236   856 root     S     856m  94%   0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 7500 -container-ip 172.18.0.2 -container-port 7500
 1305   869 root     S     855m  94%   0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/e28ca063c35280acdbb4f44f40c8917d827978a9fd2664c8119c4bb

As you can see, the difference is from an 79% to an 89% approx. of ram usage, which is around 90MB. NOTE: the Raspberry Pi 3 has 1GB or RAM, but 100MB are used for GPU memory; thats why you see 900MB in total.

We have too noticed this trend. Resource usage has been progressing in an upwards direction on all platforms with some features like multi-container support costing larger ammounts of memory.

We have an open issue & pull request with some mitigation (particularly for memory bloat) for resin-supervisor memory usage that might help you.

Hopefully the resin guys can integrate this, or improvements in this area soon…

Thank you @SplitIce, I will subscribe to the respective pull request!

hi @jcozar. Turns out the higher memory usage was to the new version of Yocto enforcing PIE for binaries which made balena also be PIE. We just released 2.15.1 that reverts that compilation option and has better memory performance. You can check the relevant PR here https://github.com/resin-os/meta-resin/pull/1166

Thank you @petrosagg ! That explains all perfectly. I am looking forward for release 2.15.1.

@jcozar Hi, pinging you just to let you know that v2.15.1 is released in production for our Fin board.

Oh that’s perfect! Thank you @majorz