If used a lot of RAM the supervisor crashes and does not recover

Product support balenaOS
1

Our team has developed a container that uses a lot of RAM unfortunately it seems like the supervisor crashes and does not recover.
Supervisor: 9.14.6
balenaOS: 2.33.0

We are working with Orange Pis, I will try to replicate this issue on other devices as well…

2

Can you enable support access and provide dashboard url for device? So that we could take a closer look.

3

Hey currently we are using openBalena

4

You can use journalctl -f -a -u resin-supervisor to see supervisor logs. Can you please provide the output?

7

Another thing to try:

Can you try enabling memory limits using mem_limit for your app container through the docker-compose file. Leaving about 300Megs for the OS/balena/supervisor as a starting point. And if you need more, you can try reducing until you get to a stable threshold perhaps?

9

@zubairlk thanks for the tips.
Set the mem_limit inside docker compose to give the supervisor 270-280MB RAM seems to work fine.

When entering balena stats on the device. It seems to not show those limits tho is that correct?

12

Hi @torben what OS version are you testing this on? I just tested on my RPi3 running balenaOS 2.41 and it correctly shows the mem_limit set to 256mb in balena stats, see below. Perhaps if you are using an older OS version its possible that this was fixed in a new balena-engine version:

CONTAINER ID        NAME                              CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
02f74f26f597        bluetooth-audio_1449813_1045053   0.00%               9.734MiB / 256MiB     3.80%               0B / 0B             23.9MB / 12.3kB     12
15f5d2b8252e        resin_supervisor                  0.03%               32.13MiB / 475.9MiB   6.75%               0B / 0B             33.4MB / 109kB      13