Figure out cause of unexpected Balena Reboot after 'x' time

Product support
1

Hi,

I have a fleet of Raspberry Pi’s running Docker on the Balena dashboard. I have a program that constantly runs inside the Docker container. It monitors values through a sensor and outputs them to the logs.

After ‘x’ amount of time (x varies), the device goes into heartbeat mode, the diagnostic says that the device rebooted, and then the logs print NAN after the device is back online. My thread that is running, dies as well. Unsure of what is causing it.

I have looked through the device diagnostics and couldn’t find anything. I am pretty new to Balena. Could someone help me understand what’s causing it to reboot? The reboot occurred between 17:45 and 17:48pm UTC. I am happy to send over the diagnostic log files if required.

2

Hi @dfalt,
Please share any log files you have and also enable persistent logging (Configuration - Balena Documentation). Without any other info, I’m wondering if it might be a powering issue. What power supply are you using and do your sensors require a lot of power? Also what type of network is your device on?

4

Diagnostic files for 2 different instances it has done this:
17:45 timestamp
0b9c38964441624f6a435dc2864b8864_diagnostics_2022.02.15_18.00.27+0000.txt (545.4 KB)
20:37 timestamp
0b9c38964441624f6a435dc2864b8864_diagnostics_2022.02.14_21.01.09+0000.txt (542.9 KB)

Log Files:
17:45 time stamp
fluorometer-server (3).log (248.1 KB)
20:37 time stamp
fluorometer-server (2).log (62.5 KB)

So yesterday after posting this query, I finally managed to reproduce the issue. I unplugged the PI, and plugged it back in. This caused the same error. The container itself didn’t restart correctly? I’m not sure what the actual process Balena has implemented for unplugging and plugging in is.

I did enable persistent logging through the dashboard. For some reason a var/log/journal directory does not exist on this device. I have my own logging software so I collect logs through that.

So Pi is connected to a Relay which is in turn connected to an LCD screen. The relay takes in a sensor and 2 valves. They all have their own power supplies.

Pi 4 - 20 W USB power supply with USB Cable rated for 5 A
Sensor - 12 VDC, draws 30 mA
Valve, 24 VDC, draws 0.25 A, running four valves (total draw 1 A)

Relay hat specs:

  • relays 24 VDC/6A
  • ADC input voltage 0-3.3 V [this is the range of voltage the sensor can output]

Network is a WIFI-2.4ghz network.

5

The interesting part is that in the case of a power down, the docker container does not restart. The Pi has absolutely no power and when it comes back online, the container does not restart.

6

Hi @Dfalt can you share the Dockerfile for the fluoremeter server? I’ll poke around to see what the best starting/restarting protocol is.