jcozar
August 23, 2018, 5:43pm
1
Hi,
I am analysing the memory usage of my devices, and I have noticed an increment of memory usage just updating the resin OS version. The difference that I am talking about is around a 10% in a RPI3 (1GB or RAM), comparing:
Oldest image: VERSION HOST OS VERSION Resin OS 2.12.7+rev1; SUPERVISOR VERSION 7.4.3
Is this a normal or expected behaviour? Might it be configured through some fleet env vars or something? My problem is that I am using around 90% or RAM and sometimes it throws an OOM signal which reboot my devices.
Thank you in advance!
imrehg
August 23, 2018, 5:57pm
4
Hi, where do you see the larger memory usage, your application containers use more memory, or the supervisor, or something else?
We have some example projects that we run to monitor device resource use, and would like to replicate what you see.
The main big change between those versions is that the newer resinOS version is running a newer balena version.
jcozar
August 23, 2018, 6:15pm
5
The application containers use exactly the same memory usage percentage (the image for those applications are the same in both devices). I am using top to check that and I am not sure about the rest. I attach the output of top command in oldest and newest version, ordering processes by memory usage (the rest are basically the application containers which use exactly the same amount of RAM):
Oldest:
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
767 1 root S 1254m 138% 0% /usr/bin/balenad --experimental --log-driver=journald -s aufs -H fd:// --dns 10.114.102.1 --bip 10.114.101.1/24 --fixed-cidr=10.114.101.0/25
783 767 root S 1002m 111% 0% balena-containerd -l unix:///var/run/balena/libcontainerd/balena-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/balena/libcontainerd/containerd
700 662 root S 914m 101% 0% balena-containerd -l unix:///var/run/balena-host/libcontainerd/balena-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/balena-host/libcontainerd/
662 1 root S 898m 99% 0% /usr/bin/balenad --delta-data-root=/mnt/sysroot/active/balena --delta-storage-driver=aufs --log-driver=journald -s aufs --data-root=/mnt/sysroot/inactive/balena -H unix:///v
2785 783 root S 873m 96% 0% balena-containerd-shim 5c23ebd688e92f880baa580b661b69fc1633f83a139f2303b172b82c011b8f84 /var/run/balena/libcontainerd/5c23ebd688e92f880baa580b661b69fc1633f83a139f2303b172b82
21999 783 root S 873m 96% 0% balena-containerd-shim a9085da0250041623fe976151b0d1b2eed4d5539e8cf1bf453f54f7f307b96bf /var/run/balena/libcontainerd/a9085da0250041623fe976151b0d1b2eed4d5539e8cf1bf453f54f7
21291 783 root S 865m 96% 0% balena-containerd-shim a4f5331324e17309a2d8d05b9517a726ac6ae5f4196054620e1f40683d168417 /var/run/balena/libcontainerd/a4f5331324e17309a2d8d05b9517a726ac6ae5f4196054620e1f406
21300 783 root S 865m 96% 0% balena-containerd-shim 9d74393d9437270f39907a6a52bc9533ff013f80e5af35eb10d6a5553e3b564e /var/run/balena/libcontainerd/9d74393d9437270f39907a6a52bc9533ff013f80e5af35eb10d6a55
22412 783 root S 865m 96% 0% balena-containerd-shim a6c02c703dfc20f038e3ccf17628457799c40f6145c532edf372bde02c3ce686 /var/run/balena/libcontainerd/a6c02c703dfc20f038e3ccf17628457799c40f6145c532edf372bde
22153 783 root S 865m 96% 0% balena-containerd-shim 8ffa6a15927ddaa6ccce6c57be3e08cfe7fc8ac9cd4a71ee872e1fdbe091b3df /var/run/balena/libcontainerd/8ffa6a15927ddaa6ccce6c57be3e08cfe7fc8ac9cd4a71ee872e1fd
21137 783 root S 865m 96% 0% balena-containerd-shim 8b4632fc98af9fc43a9b4906cc549c86f4be1b75dd74600beb735ad4bcc042a2 /var/run/balena/libcontainerd/8b4632fc98af9fc43a9b4906cc549c86f4be1b75dd74600beb735ad
2776 2732 root S 865m 96% 0% balena start --attach resin_supervisor
21690 783 root S 865m 96% 0% balena-containerd-shim e6bc9fdacf17bab67e60fecdcce2e356a68c68fc896ae67d6d314b138baa72fb /var/run/balena/libcontainerd/e6bc9fdacf17bab67e60fecdcce2e356a68c68fc896ae67d6d314b1
21427 783 root S 865m 96% 0% balena-containerd-shim 0fa1da58d1cc4fc9bc398a41cd4ca561bb3a25e2323ba765399e08a1e1de42f1 /var/run/balena/libcontainerd/0fa1da58d1cc4fc9bc398a41cd4ca561bb3a25e2323ba765399e08a
22257 783 root S 865m 96% 0% balena-containerd-shim f1f4f298840269bd03b36a7f04b0bcdf1bb69fa91d81f0ffd5f09abcf5d13863 /var/run/balena/libcontainerd/f1f4f298840269bd03b36a7f04b0bcdf1bb69fa91d81f0ffd5f09ab
21035 783 root S 865m 96% 0% balena-containerd-shim 25f61cfab19b9f971d6fdb3f3b1019bb2c8926e70f843f58b3fedec0f6f7b139 /var/run/balena/libcontainerd/25f61cfab19b9f971d6fdb3f3b1019bb2c8926e70f843f58b3fedec
21282 767 root S 857m 95% 0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 7500 -container-ip 172.18.0.4 -container-port 7500
21311 767 root S 857m 95% 0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.18.0.5 -container-port 80
2802 2785 root S 137m 15% 0% node /usr/src/app/dist/app.js
Newest:
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
849 1 root S 1337m 148% 0% /usr/bin/balenad --experimental --log-driver=journald -s aufs -H fd:// --dns 10.114.102.1 --bip 10.114.101.1/24 --fixed-cidr=10.114.101.0/25
863 849 root S 906m 100% 0% balena-containerd --config /var/run/balena/containerd/containerd.toml
726 688 root S 905m 100% 0% balena-containerd --config /var/run/balena-host/containerd/containerd.toml
688 1 root S 897m 99% 0% /usr/bin/balenad --delta-data-root=/mnt/sysroot/active/balena --delta-storage-driver=aufs --log-driver=journald -s aufs --data-root=/mnt/sysroot/inactive/balena -H unix:///v
2554 863 root S 873m 96% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/9dc3ffe9f817418080537f9f818ab80495194974d40d6ccf09371af
3366 3340 root S 873m 96% 0% balena start --attach resin_supervisor
3377 863 root S 865m 95% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/c6c70e7fa6f82f192b101b56d04774af73fbebb1e37e8e5467f3775
2921 863 root S 865m 95% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/5cd5b3cea2ee2f480983168a8133c6b88e105227d88704a1707c10d
2339 863 root S 865m 95% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/09969f35b064680896f946bcd8d9817f7637e5154080896c65a72f3
3004 863 root S 864m 95% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/e28ca063c35280acdbb4f44f40c8917d827978a9fd2664c8119c4bb
2560 863 root S 864m 95% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/2457b45f1494f011d0c58562614d6e3ab7f59e3c2ac4a95f09be279
2036 849 root S 864m 95% 0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 7500 -container-ip 172.18.0.6 -container-port 7500
2500 863 root S 856m 95% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/0db9a2b926e7336edb57965dbdb83e31bc9601acd94239deb6f486b
2102 863 root S 855m 94% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/1cb5a5eb6061994fe3137d48a652339cd17181304e6984fce5b31ea
2421 863 root S 855m 94% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/0732265d6523659e547fb490f16a9526835fdc58b3746eeea59f066
3393 3377 root S 136m 15% 0% node /usr/src/app/dist/app.js
Just to give more information, I have not tried to swap SD cards between these devices. I can try that in order to discard different hardware, or just hardware problems.
@jcozar , the first thing that comes to mind is that in resinOS 2.14.3 balena was upgraded to v17.12 which is obviously based on docker 17.12. My guess is that new changes in balena/docker probably increased the RAM usage, but perhaps @petrosagg can speak a bit more to if this could be the case
jcozar
August 25, 2018, 9:49am
7
I swapped the SD cards between both devices and it happens the same with the OS versions, so there is not a hardware problem.
I have configured both devices exactly in the same way. Again, I report you the output of the top command for the oldest and the newest OS versions:
Oldest:
Mem: 728704K used, 196484K free, 12944K shrd, 42672K buff, 252804K cached
CPU: 4% usr 1% sys 0% nic 94% idle 0% io 0% irq 0% sirq
Load average: 0.07 0.11 0.15 2/402 23726
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
748 1 root S 1186m 131% 0% /usr/bin/balenad --experimental --log-driver=journald -s aufs -H fd:// --dns 10.114.102.1 --bip 10.114.101.1/24 --fixed-cidr=10.114.101.0/25
779 748 root S 986m 109% 0% balena-containerd -l unix:///var/run/balena/libcontainerd/balena-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/balena/libcontainerd/containerd
672 1 root S 898m 99% 0% /usr/bin/balenad --delta-data-root=/mnt/sysroot/active/balena --delta-storage-driver=aufs --log-driver=journald -s aufs --data-root=/mnt/sysroot/inactive/balena -H unix:///v
710 672 root S 898m 99% 0% balena-containerd -l unix:///var/run/balena-host/libcontainerd/balena-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/balena-host/libcontainerd/
2471 779 root S 873m 96% 0% balena-containerd-shim 5c23ebd688e92f880baa580b661b69fc1633f83a139f2303b172b82c011b8f84 /var/run/balena/libcontainerd/5c23ebd688e92f880baa580b661b69fc1633f83a139f2303b172b82
3474 779 root S 873m 96% 0% balena-containerd-shim cffd937de4d2b3f0529085bfcee564b4460cd240992cf9e716096b5841abf091 /var/run/balena/libcontainerd/cffd937de4d2b3f0529085bfcee564b4460cd240992cf9e716096b5
3407 779 root S 873m 96% 0% balena-containerd-shim 059770f4fc5087964da8444eecf4f6817afb6feb7ccc4d81a6f6e974bfb1fc92 /var/run/balena/libcontainerd/059770f4fc5087964da8444eecf4f6817afb6feb7ccc4d81a6f6e97
3681 779 root S 873m 96% 0% balena-containerd-shim 6f31ce31077e15e6a758581971e33eb39a568dda8033c492e739b11a49645efa /var/run/balena/libcontainerd/6f31ce31077e15e6a758581971e33eb39a568dda8033c492e739b11
3470 779 root S 873m 96% 0% balena-containerd-shim 394b48afbf50ec9bde961b49ea07ec3ba8338cdda73caa53063129c982cf85a0 /var/run/balena/libcontainerd/394b48afbf50ec9bde961b49ea07ec3ba8338cdda73caa53063129c
3134 779 root S 866m 96% 0% balena-containerd-shim c6dc92911963fffe619bbc5f848895cec15bb7efa1d67d2d64d579794f5b77de /var/run/balena/libcontainerd/c6dc92911963fffe619bbc5f848895cec15bb7efa1d67d2d64d5797
3290 779 root S 865m 96% 0% balena-containerd-shim a2614ecaba19bcf885ac804ace0dd13ba063bbe55f0e534128490c862a0bafc6 /var/run/balena/libcontainerd/a2614ecaba19bcf885ac804ace0dd13ba063bbe55f0e534128490c8
3476 779 root S 865m 96% 0% balena-containerd-shim c0c9fae9c5c1893c7a465532f9c111a5501317885d537906ddef6171f60454fd /var/run/balena/libcontainerd/c0c9fae9c5c1893c7a465532f9c111a5501317885d537906ddef617
2462 2434 root S 865m 96% 0% balena start --attach resin_supervisor
3767 779 root S 865m 96% 0% balena-containerd-shim 5b300662c2481af2ff34f7a61afbdbf46de49b9a9074ef6c783920d954bbe207 /var/run/balena/libcontainerd/5b300662c2481af2ff34f7a61afbdbf46de49b9a9074ef6c783920d
3786 779 root S 865m 95% 0% balena-containerd-shim 122a9fb280e8e3addd5135c463d5da77a0ff32188cf3fcf97200b6dc82c7e085 /var/run/balena/libcontainerd/122a9fb280e8e3addd5135c463d5da77a0ff32188cf3fcf97200b6d
3367 748 root S 857m 95% 0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.19.0.4 -container-port 80
3275 748 root S 857m 95% 0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 7500 -container-ip 172.19.0.3 -container-port 7500
Newest:
Mem: 825696K used, 98328K free, 7280K shrd, 11336K buff, 64500K cached
CPU: 3% usr 0% sys 0% nic 95% idle 0% io 0% irq 0% sirq
Load average: 0.19 0.19 0.18 2/338 24528
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
856 1 root S 1139m 126% 0% /usr/bin/balenad --experimental --log-driver=journald -s aufs -H fd:// --dns 10.114.102.1 --bip 10.114.101.1/24 --fixed-cidr=10.114.101.0/25
869 856 root S 906m 100% 0% balena-containerd --config /var/run/balena/containerd/containerd.toml
688 1 root S 905m 100% 0% /usr/bin/balenad --delta-data-root=/mnt/sysroot/active/balena --delta-storage-driver=aufs --log-driver=journald -s aufs --data-root=/mnt/sysroot/inactive/balena -H unix:///v
728 688 root S 905m 100% 0% balena-containerd --config /var/run/balena-host/containerd/containerd.toml
2246 869 root S 873m 96% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/c6c70e7fa6f82f192b101b56d04774af73fbebb1e37e8e5467f3775
2234 2203 root S 873m 96% 0% balena start --attach resin_supervisor
1469 869 root S 865m 95% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/5cd5b3cea2ee2f480983168a8133c6b88e105227d88704a1707c10d
1260 869 root S 865m 95% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/09969f35b064680896f946bcd8d9817f7637e5154080896c65a72f3
1285 869 root S 865m 95% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/1cb5a5eb6061994fe3137d48a652339cd17181304e6984fce5b31ea
1784 869 root S 865m 95% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/ba4307233508cf87e65a722da31e036fd731e9f9898609005347ffe
1639 856 root S 865m 95% 0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.18.0.9 -container-port 80
1593 869 root S 856m 95% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/2457b45f1494f011d0c58562614d6e3ab7f59e3c2ac4a95f09be279
1570 869 root S 856m 94% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/0732265d6523659e547fb490f16a9526835fdc58b3746eeea59f066
1438 869 root S 856m 94% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/9dc3ffe9f817418080537f9f818ab80495194974d40d6ccf09371af
1337 869 root S 856m 94% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/0db9a2b926e7336edb57965dbdb83e31bc9601acd94239deb6f486b
1236 856 root S 856m 94% 0% /usr/bin/balena-proxy -proto tcp -host-ip 0.0.0.0 -host-port 7500 -container-ip 172.18.0.2 -container-port 7500
1305 869 root S 855m 94% 0% balena-containerd-shim -namespace moby -workdir /var/lib/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/e28ca063c35280acdbb4f44f40c8917d827978a9fd2664c8119c4bb
As you can see, the difference is from an 79% to an 89% approx. of ram usage, which is around 90MB. NOTE: the Raspberry Pi 3 has 1GB or RAM, but 100MB are used for GPU memory; thats why you see 900MB in total.
We have too noticed this trend. Resource usage has been progressing in an upwards direction on all platforms with some features like multi-container support costing larger ammounts of memory.
We have an open issue & pull request with some mitigation (particularly for memory bloat) for resin-supervisor memory usage that might help you.
Hopefully the resin guys can integrate this, or improvements in this area soon…
jcozar
August 27, 2018, 6:34am
9
Thank you @SplitIce , I will subscribe to the respective pull request!
hi @jcozar . Turns out the higher memory usage was to the new version of Yocto enforcing PIE for binaries which made balena also be PIE. We just released 2.15.1 that reverts that compilation option and has better memory performance. You can check the relevant PR here https://github.com/resin-os/meta-resin/pull/1166
jcozar
August 31, 2018, 7:13am
11
Thank you @petrosagg ! That explains all perfectly. I am looking forward for release 2.15.1.
majorz
September 10, 2018, 5:13am
13
@jcozar Hi, pinging you just to let you know that v2.15.1 is released in production for our Fin board.
jcozar
September 10, 2018, 7:21am
14
Oh that’s perfect! Thank you @majorz