Random Supervisor crashes on Balena OS 2.36

Product support balenaOS
1

I am facing some random crashes on Balena’s OS 2.36 Supervisor.

After the OS update, the app started normally but after a certain amount of time under its operation regime, it stalls. No more logs are prompted and afterwards supervisor restarts, as well ass the whole application.

I reproduced this behaviour, this time having a look at the “journalctl” and did a dump of the logs.

root@b701352:~# journalctl -f -n 1000 -u resin-supervisor
-- Logs begin at Thu 2019-06-13 11:00:47 UTC. --
Jun 13 11:04:57 b719252 resin-supervisor[32285]: Supervisor API: GET /v1/healthy 200 - 6.683 ms
Jun 13 11:09:58 b719252 resin-supervisor[32285]: Supervisor API: GET /v1/healthy 200 - 9.613 ms
Jun 13 11:14:58 b719252 resin-supervisor[32285]: Supervisor API: GET /v1/healthy 200 - 5.215 ms
Jun 13 11:19:59 b719252 resin-supervisor[32285]: Supervisor API: GET /v1/healthy 200 - 6.904 ms
Jun 13 11:22:37 b719252 resin-supervisor[32285]: (node:1) UnhandledPromiseRejectionWarning: TimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at /usr/src/app/dist/app.js:21:42906
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at c (/usr/src/app/dist/app.js:9:74428)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at /usr/src/app/dist/app.js:103:29838
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at c (/usr/src/app/dist/app.js:9:74428)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at A._settlePromiseFromHandler (/usr/src/app/dist/app.js:293:114147)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at A._settlePromise (/usr/src/app/dist/app.js:293:114947)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at A._settlePromise0 (/usr/src/app/dist/app.js:293:115646)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at A._settlePromises (/usr/src/app/dist/app.js:293:116887)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at f (/usr/src/app/dist/app.js:293:118592)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at p (/usr/src/app/dist/app.js:293:118531)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at s._drainQueues (/usr/src/app/dist/app.js:293:120051)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:293:118273)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at runCallback (timers.js:705:18)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at tryOnImmediate (timers.js:676:5)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at processImmediate (timers.js:658:5)
Jun 13 11:22:37 b719252 resin-supervisor[32285]: (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 25)
Jun 13 11:22:37 b719252 resin-supervisor[32285]: (node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Jun 13 11:22:37 b719252 resin-supervisor[32285]: (node:1) UnhandledPromiseRejectionWarning: TimeoutError: Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at /usr/src/app/dist/app.js:21:42906
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at c (/usr/src/app/dist/app.js:9:74428)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at /usr/src/app/dist/app.js:103:29838
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at c (/usr/src/app/dist/app.js:9:74428)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at A._settlePromiseFromHandler (/usr/src/app/dist/app.js:293:114147)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at A._settlePromise (/usr/src/app/dist/app.js:293:114947)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at A._settlePromise0 (/usr/src/app/dist/app.js:293:115646)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at A._settlePromises (/usr/src/app/dist/app.js:293:116887)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at f (/usr/src/app/dist/app.js:293:118592)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at p (/usr/src/app/dist/app.js:293:118531)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at s._drainQueues (/usr/src/app/dist/app.js:293:120051)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:293:118273)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at runCallback (timers.js:705:18)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at tryOnImmediate (timers.js:676:5)
Jun 13 11:22:37 b719252 resin-supervisor[32285]:     at processImmediate (timers.js:658:5)

As you can see, at the biginning the API responds as healthy with a 200. But afterwards those “UnhandledPromiseRejectionWarning” appear.

I would really appreciate any support on this.
Best regards.

6

Hello @snorkman and welcome to the balena community!

Thank you for reporting this crash, we very much appreciate it! We have actually already fixed this issue and it should be released in an OS update soon (specifically supervisor v9.15.6 and balenaOS v2.37.0). There’s more background in this forum thread, but the crash is likely a result of the logging frequency for your application.

We will update you when that OS version becomes available if you let us know what device type you are using such that you can upgrade and pick up the fix.

8

Thanks for the quick reply :slight_smile:
I am using a RPi 3. Any estimated release date for this OS update?

11

Hi again @snorkman,

We expect to release balenaOS v2.38 which contains the changes sometime this week, tentatively targeting Wednesday. We will update you when that has been released.

Thank you again for your report!

1 Like