Container ssh keeps disconnecting and reconnecting every 6 minutes

I am running a docker container with balenaOS on a Raspberry Pi Compute Module 4. I am ssh’ed to the container through the terminal on balenaCloud, however it will consistently disconnect every 6 minutes.

When disconnecting, the following messages are displayed:

root@5640dcd:/home/dev# Cannot connect to the balenaEngine daemon at unix:///var/run/balena-engine.sock. Is the balenaEngine daemon running?
SSH session disconnected
SSH reconnecting…
Spawning shell…
root@5640dcd:/home/dev#

Occasionally, but not with every disconnect and reconnection, “Supervisor starting” will be output in the Logs.

I noticed that when I’m connected to the Host OS on a different terminal window (also on balenaCloud), my connection is not dropped when my connection to the container is. Maybe the service is being restarted every so often, however I don’t see any logs indicating that the service is killed and restarted. But what might be the cause for this and how might I fix it?

The balenaOS version is balenaOS 2.115.1
The supervisor version is 14.11.1

The device in question is here: balena dashboard
I have enabled support access for one week.

I’ve just tried running balena container wait _____ on the Host OS, and when the connection is dropped I get the following exit code:
137

I have also run a device health check, where there was a failure for check_container_engine:
Some container_engine issues detected:
test_container_engine_restarts Container engine balena has 3 restarts and may be crashlooping (most recent start time: Fri 2023-08-04 22:56:39 UTC)

I just reinstalled balenaOS on the Pi, which seems to have fixed the issue. If anyone knows how to stop the crash loop without having to completely reflash the Pi that would be great, but this is my solution for now.

Hello @ez1234 are you running into this issue on more than one device? I’m assuming you re-flashed the device that you shared support access for? If so, it would have a new UUID and link, but any logs that could be used for troubleshooting the original issue would be gone. If you do see this again, please enable persistent logging and then provide access to the device.

Yep I re-flashed the device and the issue hasn’t happened again (yet). If I do see this issue again I’ll make sure to follow those steps, thanks!

1 Like