Find reason for automatic reboots in Balena OS

r.krucker · October 24, 2023, 8:20am

Hi,

We have an OpenBalena server running and using Asus Tinker Board S devices. The current BalenaOS image for that device is BalenaOS 2.38.3+rev1. Occasionally, our application running on the fleet, is unexpectedly restarting or maybe the whole device. Due to our application has a detach function that does some cleanups and writes some logs before stopping, we are able to know, that the container gets a SIGTERM signal. But we are not able to find out, who is sending the SIGTERM signal (maybe the Supervisor Service) and why the signal was sent.

We are using even other devices types to run the same application (Raspberry PI 2 and Revolution PI Core 3), but we never had an identical issue at one of these devices.

Could it be possible, that the old version of Supervisor (9.15.7), provided by the BalenaOS image for Asus Tinker Board, cause that problem?

r.krucker · November 24, 2023, 9:23am

Fortunately, I was able to analyse a case just few minutes after it happened. Therefore I realized, that not the Balena OS reboots, but our applicaton, running on it, has restartet. In the journal of the device, I found a huge amount of messages from “balenad”:

Nov 24 07:48:06 d2826e7 c4cbeb3ef8e0[739]: [90B blob data]
Nov 24 07:48:15 d2826e7 ebd8ee3c3e57[739]: Supervisor API: GET /v1/healthy 200 - 3.495 ms
Nov 24 07:48:15 d2826e7 resin-supervisor[1240]: Supervisor API: GET /v1/healthy 200 - 3.495 ms
Nov 24 07:48:15 d2826e7 balenad[739]: fatal error: unexpected signal during runtime execution
Nov 24 07:48:15 d2826e7 balenad[739]: [signal SIGSEGV: segmentation violation code=0x2 addr=0xb6f8e920 pc=0xb6e1f360]
Nov 24 07:48:15 d2826e7 balenad[739]: runtime stack:
Nov 24 07:48:15 d2826e7 balenad[739]: runtime.throw(0x12a35da, 0x2a)
Nov 24 07:48:15 d2826e7 balenad[739]:         /usr/lib/go/src/runtime/panic.go:616 +0x60
Nov 24 07:48:15 d2826e7 balenad[739]: runtime.sigpanic()
Nov 24 07:48:15 d2826e7 balenad[739]:         /usr/lib/go/src/runtime/signal_unix.go:372 +0x22c
Nov 24 07:48:15 d2826e7 balenad[739]: goroutine 321 [syscall]:
Nov 24 07:48:15 d2826e7 balenad[739]: runtime.cgocall(0xf27fac, 0x13ffe760, 0x129f6d4)
Nov 24 07:48:15 d2826e7 balenad[739]:         /usr/lib/go/src/runtime/cgocall.go:128 +0x64 fp=0x13ffe740 sp=0x13ffe728 pc=0x155d8
Nov 24 07:48:15 d2826e7 balenad[739]: github.com/docker/docker/daemon/logger/journald._Cfunc_wait_for_data_cancelable(0x34f85a8, 0x33, 0x0)
Nov 24 07:48:15 d2826e7 balenad[739]:         _cgo_gotypes.go:397 +0x38 fp=0x13ffe75c sp=0x13ffe740 pc=0xe4a810
Nov 24 07:48:15 d2826e7 balenad[739]: github.com/docker/docker/daemon/logger/journald.(*journald).followJournal.func1.1(0x34f85a8, 0x33, 0x34f85a8)
Nov 24 07:48:15 d2826e7 balenad[739]:         /yocto/resin-board/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/balena/18.09.6-dev+git95c7371304f9cef494efe93f0a8ffd53a75eac21-r0/git/src/import/.gopath/src/github.com/docker/docker/daemon/logger/journald/read.go:272 >
Nov 24 07:48:15 d2826e7 balenad[739]: github.com/docker/docker/daemon/logger/journald.(*journald).followJournal.func1(0x34f85a8, 0x33, 0x3b, 0x13d6f360, 0x13db6690, 0x13a09658, 0x0, 0x0, 0x13f91640)
Nov 24 07:48:15 d2826e7 balenad[739]:         /yocto/resin-board/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/balena/18.09.6-dev+git95c7371304f9cef494efe93f0a8ffd53a75eac21-r0/git/src/import/.gopath/src/github.com/docker/docker/daemon/logger/journald/read.go:272 >
Nov 24 07:48:15 d2826e7 balenad[739]: runtime.goexit()
Nov 24 07:48:15 d2826e7 balenad[739]:         /usr/lib/go/src/runtime/asm_arm.s:1015 +0x4 fp=0x13ffe7c4 sp=0x13ffe7c4 pc=0x7485c
Nov 24 07:48:15 d2826e7 balenad[739]: created by github.com/docker/docker/daemon/logger/journald.(*journald).followJournal
Nov 24 07:48:15 d2826e7 balenad[739]:         /yocto/resin-board/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/balena/18.09.6-dev+git95c7371304f9cef494efe93f0a8ffd53a75eac21-r0/git/src/import/.gopath/src/github.com/docker/docker/daemon/logger/journald/read.go:268 >
Nov 24 07:48:15 d2826e7 balenad[739]: goroutine 1 [chan receive, 48765 minutes]:
Nov 24 07:48:15 d2826e7 balenad[739]: github.com/docker/docker/cmd/dockerd.(*DaemonCli).start(0x13b92220, 0x13aeccf0, 0x0, 0x0)
Nov 24 07:48:15 d2826e7 balenad[739]:         /yocto/resin-board/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/balena/18.09.6-dev+git95c7371304f9cef494efe93f0a8ffd53a75eac21-r0/git/src/import/.gopath/src/github.com/docker/docker/cmd/dockerd/daemon.go:222 +0x878
Nov 24 07:48:15 d2826e7 balenad[739]: github.com/docker/docker/cmd/dockerd.runDaemon(0x13aeccf0, 0x13aea300, 0x0)
Nov 24 07:48:15 d2826e7 balenad[739]:         /yocto/resin-board/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/balena/18.09.6-dev+git95c7371304f9cef494efe93f0a8ffd53a75eac21-r0/git/src/import/.gopath/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:7 +0x30
Nov 24 07:48:15 d2826e7 balenad[739]: github.com/docker/docker/cmd/dockerd.newDaemonCommand.func1(0x13b088c0, 0x13aea500, 0x0, 0x10, 0x0, 0x0)
Nov 24 07:48:15 d2826e7 balenad[739]:         /yocto/resin-board/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/balena/18.09.6-dev+git95c7371304f9cef494efe93f0a8ffd53a75eac21-r0/git/src/import/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:29 +0x50
Nov 24 07:48:15 d2826e7 balenad[739]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0x13b088c0, 0x138ac098, 0x10, 0x11, 0x13b088c0, 0x138ac098)
Nov 24 07:48:15 d2826e7 balenad[739]:         /yocto/resin-board/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/balena/18.09.6-dev+git95c7371304f9cef494efe93f0a8ffd53a75eac21-r0/git/src/import/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/comma>
Nov 24 07:48:15 d2826e7 balenad[739]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x13b088c0, 0x13feea0, 0x1055830, 0x13feea8)
Nov 24 07:48:15 d2826e7 balenad[739]:         /yocto/resin-board/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/balena/18.09.6-dev+git95c7371304f9cef494efe93f0a8ffd53a75eac21-r0/git/src/import/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/comma>
Nov 24 07:48:15 d2826e7 balenad[739]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(0x13b088c0, 0x138a60d8, 0x125131d)
Nov 24 07:48:15 d2826e7 balenad[739]:         /yocto/resin-board/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/balena/18.09.6-dev+git95c7371304f9cef494efe93f0a8ffd53a75eac21-r0/git/src/import/.gopath/src/github.com/docker/docker/vendor/github.com/spf13/cobra/comma>
Nov 24 07:48:15 d2826e7 balenad[739]: github.com/docker/docker/cmd/dockerd.Main()
Nov 24 07:48:15 d2826e7 balenad[739]:         /yocto/resin-board/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/balena/18.09.6-dev+git95c7371304f9cef494efe93f0a8ffd53a75eac21-r0/git/src/import/.gopath/src/github.com/docker/docker/cmd/dockerd/docker.go:71 +0x7c
...

BalenaDeviceJournal.log (174.1 KB)

Any ideas?