Hi all, As the title suggests, last week one of my devices (Raspberry Pi 3B+) running BalenaOS (2.46.1+rev1) failed. This has left me with a stacktrace of about 1000 lines ( balenad_crash.log (149.6 KB) ). Of course, I would like to prevent this from happening in the future, so I was wondering if …

Hi there, in order to be able to help you I will need the following informations: what OS version are you running, and what are the modifications you made to it? is this device on balenaCloud? if so, could you share the device uuid and grant support access to it? regarding multi-threading, …

Hello @curcuz , Thank you for reply. I am running BalenaOS v2.46.1+rev1 with a custom script in /etc/NetworkManager/dispatcher.d; this script is using for enabling Roaming on certain Huawei 4G modems when they are plugged in. I have also modified /etc/systemd/journald.conf.d/journald-balena-os.c…

thanks, this is already good info to help investigating. I’m looking into alternative ways to achieve the configuration you currently perform as a balenaOS modification to see if you could rely on stock balenaOS and try newer releases

Hi there – one additional bit of detail: you can check the maximum number of threads like so: cat /proc/sys/kernel/threads-max and you can count the number of open threads by running this command : cat /proc/*/status | awk '/Threads/ {total += $NF} END {print "Total threads: ", total}' Note that …

Thank you for your suggestions, @saintaardvark . As far as I can tell, this has only happened that one time. Right now the application appears to be running nominal, and the commands you gave suggest 243/15369 threads (total/max). I will see if I can move the modem configuration script to be trigg…

Hi @TJvV – thanks for the additional information. That sounds like there was a lot of headroom at the moment you checked. I would definitely consider monitoring this, though. I’ve created a bug here ; feel free to subscribe to that issue if you’d like updates. We’re looking into the backtrace, an…

Hi, Just this morning it occurred again. I have attached the stack trace together with the last few system messages. balenad_crash_2020_11_02.log (150.9 KB) At the time of crashing my application did not report any warnings/errors. I sadly do not have any monitoring of the total number of threa…

Hi, thanks for the stack trace. This looks to be the same problem we have been debugging before. In summary, there are two things that can cause it: Reaching the maximum number of threads. For a balena application container this is actually around 4915. Insufficient memory (less likely) As it see…

Does Balena have intention on raising the 4915 threads limit? Or making it customizable?

Hey @krix , could you please elaborate about the problem you are trying to resolve? Did you hit the 4915 threads limit? Thanks!

balenaOS 2.46.1+rev1 crashed with pthread_create failed: Resource temporarily unavailable

Product support

krix August 5, 2021, 2:16pm 45

I have raised a question about the exact requirements in a new thread: Changing the allowed maximum number of threads

Topic		Replies	Views
Balena Crash / Balena Reliability Product support	13	1223	October 30, 2018
High load / CPU usage and container stopping Product support	10	1513	October 17, 2019
balenad[PID]: runtime/cgo: pthread_create failed: Resource temporarily unavailable Product support support	10	1604	December 15, 2020
An average user CPU usage of 35% is reported on my raspberry pi which cannot directly be attributed to any container balenaOS raspberrypi3	14	4734	May 29, 2019
Container fails to start - "failed to attach 1 to compat systemd cgroup" balenaOS	40	6371	September 22, 2019

balenaOS 2.46.1+rev1 crashed with pthread_create failed: Resource temporarily unavailable

Related topics