Supervisor continuously restarts - major outage of Balena

Product support balenaOS
1

Last night my fleet got bricked because the supervisor kept restarting … I thought it was something in my code or config variables, so I started a new fleet - same issue.

I even deleted my balena account and started a fresh one. Same issue. Even before pushing any code to the devices, the default OS just continuously restarts the supervisor.

It seems like this is a major outage of Balena.

here’s a sample of the logs from a fresh fleet with a fresh device, no code has been pushed:

2024-07-22T09:13:36-05:00 Supervisor starting
2024-07-22T09:13:42-05:00 Applying configuration change {“SUPERVISOR_POLL_INTERVAL”:“900000”,“SUPERVISOR_DELTA”:“1”,“SUPERVISOR_DELTA_VERSION”:“3”}
2024-07-22T09:13:42-05:00 Applied configuration change {“SUPERVISOR_POLL_INTERVAL”:“900000”,“SUPERVISOR_DELTA”:“1”,“SUPERVISOR_DELTA_VERSION”:“3”}
2024-07-22T09:13:44-05:00 Creating network ‘default’
2024-07-22T09:13:44-05:00 Creating volume ‘resin-data’
2024-07-22T09:13:44-05:00 Creating network ‘default’
2024-07-22T09:13:36-05:00 Supervisor starting
2024-07-22T09:13:42-05:00 Applying configuration change {“SUPERVISOR_POLL_INTERVAL”:“900000”,“SUPERVISOR_DELTA”:“1”,“SUPERVISOR_DELTA_VERSION”:“3”}
2024-07-22T09:13:42-05:00 Applied configuration change {“SUPERVISOR_POLL_INTERVAL”:“900000”,“SUPERVISOR_DELTA”:“1”,“SUPERVISOR_DELTA_VERSION”:“3”}
2024-07-22T09:13:44-05:00 Creating network ‘default’
2024-07-22T09:13:44-05:00 Creating volume ‘resin-data’
2024-07-22T09:13:44-05:00 Creating network ‘default’
2024-07-22T09:13:36-05:00 Supervisor starting
2024-07-22T09:13:42-05:00 Applying configuration change {“SUPERVISOR_POLL_INTERVAL”:“900000”,“SUPERVISOR_DELTA”:“1”,“SUPERVISOR_DELTA_VERSION”:“3”}
2024-07-22T09:13:42-05:00 Applied configuration change {“SUPERVISOR_POLL_INTERVAL”:“900000”,“SUPERVISOR_DELTA”:“1”,“SUPERVISOR_DELTA_VERSION”:“3”}
2024-07-22T09:13:44-05:00 Creating network ‘default’
2024-07-22T09:13:44-05:00 Creating volume ‘resin-data’
2024-07-22T09:13:44-05:00 Creating network ‘default’
2024-07-22T09:13:36-05:00 Supervisor starting
2024-07-22T09:13:42-05:00 Applying configuration change {“SUPERVISOR_POLL_INTERVAL”:“900000”,“SUPERVISOR_DELTA”:“1”,“SUPERVISOR_DELTA_VERSION”:“3”}
2024-07-22T09:13:42-05:00 Applied configuration change {“SUPERVISOR_POLL_INTERVAL”:“900000”,“SUPERVISOR_DELTA”:“1”,“SUPERVISOR_DELTA_VERSION”:“3”}
2024-07-22T09:13:44-05:00 Creating network ‘default’
2024-07-22T09:13:44-05:00 Creating volume ‘resin-data’
2024-07-22T09:13:44-05:00 Creating network ‘default’

2

Hello @jason37 apologizes for the experience suffered!

Could you please share more details of the device type that you are using and your fleet is using? What balenaOS and supervisor versions are you using?

Thanks

3

I am currently trying different versions of OS and supervisor to see if that solves the problem … but the issue originated with the latest version of each.

This is for Raspberry pi zero 2w

4

If I push a simple shell script that echos “this is a test” - here is what I see in the logs:

2024-07-22T10:48:11-05:00 main This is a test
2024-07-22T10:48:12-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:49:14-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:49:14-05:00 main This is a test
2024-07-22T10:49:16-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:50:17-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:50:17-05:00 main This is a test
2024-07-22T10:50:19-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:51:21-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:51:21-05:00 main This is a test
2024-07-22T10:51:22-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:52:24-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:52:24-05:00 main This is a test
2024-07-22T10:52:25-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:53:27-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:53:27-05:00 main This is a test
2024-07-22T10:53:28-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:54:30-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:54:30-05:00 main This is a test
2024-07-22T10:54:32-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:55:33-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:55:33-05:00 main This is a test
2024-07-22T10:55:35-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:56:37-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:56:37-05:00 main This is a test
2024-07-22T10:56:38-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:57:40-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:57:40-05:00 main This is a test
2024-07-22T10:57:41-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:58:43-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:58:43-05:00 main This is a test
2024-07-22T10:58:44-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:59:46-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:59:46-05:00 main This is a test
2024-07-22T10:59:47-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:00:49-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:00:49-05:00 main This is a test
2024-07-22T11:00:51-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:01:53-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:01:53-05:00 main This is a test
2024-07-22T11:01:54-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:02:56-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:02:56-05:00 main This is a test
2024-07-22T11:02:57-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:03:59-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:03:59-05:00 main This is a test
2024-07-22T11:04:00-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:05:02-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:05:02-05:00 main This is a test
2024-07-22T11:05:04-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:06:05-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:06:05-05:00 main This is a test
2024-07-22T11:06:07-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:07:08-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:07:08-05:00 main This is a test
2024-07-22T11:07:10-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:08:12-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:08:12-05:00 main This is a test
2024-07-22T11:08:13-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:09:15-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:09:15-05:00 main This is a test
2024-07-22T11:09:16-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:10:18-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:10:18-05:00 main This is a test
2024-07-22T11:10:19-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:11:21-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:11:21-05:00 main This is a test
2024-07-22T11:11:23-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:12:24-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:12:24-05:00 main This is a test
2024-07-22T11:12:26-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:13:28-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:13:28-05:00 main This is a test
2024-07-22T11:13:29-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:14:31-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:14:31-05:00 main This is a test
2024-07-22T11:14:32-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:39:47-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:40:49-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:40:49-05:00 main This is a test
2024-07-22T10:40:50-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:41:52-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:41:52-05:00 main This is a test
2024-07-22T10:41:53-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:42:55-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:42:55-05:00 main This is a test
2024-07-22T10:42:56-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:43:58-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:43:58-05:00 main This is a test
2024-07-22T10:44:00-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:45:01-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:45:01-05:00 main This is a test
2024-07-22T10:45:03-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:46:04-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:46:04-05:00 main This is a test
2024-07-22T10:46:06-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:47:08-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:47:08-05:00 main This is a test
2024-07-22T10:47:09-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:48:11-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:48:11-05:00 main This is a test
2024-07-22T10:48:12-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:49:14-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:49:14-05:00 main This is a test
2024-07-22T10:49:16-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:50:17-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:50:17-05:00 main This is a test
2024-07-22T10:50:19-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:51:21-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:51:21-05:00 main This is a test
2024-07-22T10:51:22-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:52:24-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:52:24-05:00 main This is a test
2024-07-22T10:52:25-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:53:27-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:53:27-05:00 main This is a test
2024-07-22T10:53:28-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:54:30-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:54:30-05:00 main This is a test
2024-07-22T10:54:32-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:55:33-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:55:33-05:00 main This is a test
2024-07-22T10:55:35-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:56:37-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:56:37-05:00 main This is a test
2024-07-22T10:56:38-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:57:40-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:57:40-05:00 main This is a test
2024-07-22T10:57:41-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:58:43-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:58:43-05:00 main This is a test
2024-07-22T10:58:44-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:59:46-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:59:46-05:00 main This is a test
2024-07-22T10:59:47-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:00:49-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:00:49-05:00 main This is a test
2024-07-22T11:00:51-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:01:53-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:01:53-05:00 main This is a test
2024-07-22T11:01:54-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:02:56-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:02:56-05:00 main This is a test
2024-07-22T11:02:57-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:03:59-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:03:59-05:00 main This is a test
2024-07-22T11:04:00-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:05:02-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:05:02-05:00 main This is a test
2024-07-22T11:05:04-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:06:05-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:06:05-05:00 main This is a test
2024-07-22T11:06:07-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:07:08-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:07:08-05:00 main This is a test
2024-07-22T11:07:10-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:08:12-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:08:12-05:00 main This is a test
2024-07-22T11:08:13-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:09:15-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:09:15-05:00 main This is a test
2024-07-22T11:09:16-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:10:18-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:10:18-05:00 main This is a test
2024-07-22T11:10:19-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:11:21-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:11:21-05:00 main This is a test
2024-07-22T11:11:23-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:12:24-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:12:24-05:00 main This is a test
2024-07-22T11:12:26-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:13:28-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:13:28-05:00 main This is a test
2024-07-22T11:13:29-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:14:31-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:14:31-05:00 main This is a test
2024-07-22T11:14:32-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’

5

Here’s what it looks like when you push code with a simple shell script that echos “this is a test”

2024-07-22T11:59:43-05:00 Releasing update locks
2024-07-22T11:59:44-05:00 This is a test
2024-07-22T11:59:46-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:59:48-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:59:48-05:00 This is a test
2024-07-22T11:59:49-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:59:51-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:59:51-05:00 This is a test
2024-07-22T11:59:53-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:59:55-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:59:55-05:00 This is a test
2024-07-22T11:59:57-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:59:59-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T11:59:59-05:00 This is a test
2024-07-22T12:00:01-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:00:04-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:00:04-05:00 This is a test
2024-07-22T12:00:06-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:00:11-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:00:11-05:00 This is a test
2024-07-22T12:00:12-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:00:20-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:00:20-05:00 This is a test
2024-07-22T12:00:22-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:00:36-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:00:36-05:00 This is a test
2024-07-22T12:00:38-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:01:05-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:01:05-05:00 This is a test
2024-07-22T12:01:06-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’

6

could this be related to the CrowdStrike outage? I just signed up for Balena a few days ago to evaluate it…

7

here is what it looks like with a simple shell script that echos “this is a test”

2024-07-22T12:23:11-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:23:11-05:00 main This is a test
2024-07-22T12:23:13-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:23:19-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:23:19-05:00 main This is a test
2024-07-22T12:23:21-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:23:30-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T12:23:31-05:00 main This is a test
2024-07-22T12:23:33-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’

8

this appears to be happening in every version of BalenaOS and Supervisor available to flash in the app, so 5.1.47 and 16.1.0 all the way up to 5.4.0+rev1 and 16.4.4

9

while I am primarily using Raspberry Pi Zero 2W, I just wanted to confirm that this error is also happening with Raspberry Pi 3 units as well.

So, it seems that if any user were to create a new fleet of Raspberry Pi Zero 2 or Raspberry Pi 3 units, the Balena supervisor will just keep restarting over and over before you can load any code.

And, if you load some code, it will execute up to 30s - 2m before the supervisor restarts again.

This is a very serious problem, Balena.

10

@jason37 no this should have no effect on us. Crowdstrike related outage was about Windows Patch that bricked the update. At balena, we are not running Crowdstrike anywhere nor do we operate windows in production.

11

Could you please run Diagnostics from balenaCloud and share the logs that you get @jason37 ? Do you see any issues with your supervisor?

In the other hand, could you please confirm if your container is programmed to get restarted all the time?

Thanks

12

Device Diagnostics uploaded: 0738a534635b26c615b63ebebf9eb195_diagnostics_2024.07.23_12.31.19+0000.txt (396.4 KB)

Device Health Check:
{“diagnose_version”:“4.23.1”,“checks”:[{“name”:“check_balenaOS”,“success”:true,“status”:“Supported balenaOS 2.x detected”},{“name”:“check_container_engine”,“success”:true,“status”:“No container_engine issues detected”},{“name”:“check_localdisk”,“success”:true,“status”:“No localdisk issues detected”},{“name”:“check_memory”,“success”:true,“status”:“53% memory available”},{“name”:“check_networking”,“success”:true,“status”:“No networking issues detected”},{“name”:“check_os_rollback”,“success”:true,“status”:“No OS rollbacks detected”},{“name”:“check_supervisor”,“success”:true,“status”:“Supervisor is running & healthy”},{“name”:“check_temperature”,“success”:true,“status”:“No temperature issues detected”},{“name”:“check_timesync”,“success”:true,“status”:“Time is synchronized”},{“name”:“check_under_voltage”,“success”:true,“status”:“No under-voltage events detected”}]}

Supervisor status:
{
“api_port”: 48484,
“ip_address”: “192.168.0.34 2603:8080:1506:fb00:c77d:109d:2af0:352d”,
“os_version”: “balenaOS 5.4.0+rev1”,
“mac_address”: “2C:CF:67:25:99:DF”,
“supervisor_version”: “16.4.1”,
“update_pending”: false,
“update_failed”: false,
“update_downloaded”: false
}
{
“0738a534635b26c615b63ebebf9eb195”: {
“name”: “rough-turnip”,
“apps”: {
“5fae038b957b440185143462558773f9”: {
“id”: 1887016,
“name”: “raspberrypi0-2w-64”,
“is_host”: true,
“class”: “app”,
“releases”: {
“f9e146632e4f0edeabaa65198912fb38”: {
“id”: 3094460,
“services”: {
“main”: {
“id”: 1326363,
“image_id”: 9259901,
“image”: “registry2.balena-cloud.com/v2/23c2a08c7e23c3c355b65a94296bec24@sha256:064b19cdb00de24de1221f3b8af36e4de474c177157e44d496066f7304bf8730”,
“environment”: {},
“labels”: {
“io.balena.image.store”: “root”,
“io.resin.features.dbus”: “1”,
“io.resin.features.firmware”: “1”,
“io.resin.features.kernel-modules”: “1”,
“io.resin.features.resin-api”: “1”,
“io.resin.features.supervisor-api”: “1”
},
“composition”: {
“tty”: true,
“image”: “sha256:c79ab1e7de6297b12b24be8d22de1cb63ac7c7ea93a41baa4fe1c45be3fadc35”,
“labels”: {
“io.resin.features.dbus”: “1”,
“io.resin.features.firmware”: “1”,
“io.resin.features.resin-api”: “1”,
“io.resin.features.kernel-modules”: “1”,
“io.resin.features.supervisor-api”: “1”
},
“restart”: “always”,
“volumes”: [
“resin-data:/data”
],
“privileged”: true,
“network_mode”: “host”
}
}
},
“networks”: {},
“volumes”: {
“resin-data”: {}
}
}
}
},
“d8dd3018d1624463a1d5cefa42321df4”: {
“id”: 2146129,
“name”: “ellie-prototype-alpha”,
“is_host”: false,
“class”: “fleet”
}
},
“config”: {
“RESIN_SUPERVISOR_DELTA_VERSION”: “3”,
“RESIN_SUPERVISOR_NATIVE_LOGGER”: “true”,
“RESIN_HOST_CONFIG_avoid_warnings”: “1”,
“RESIN_HOST_CONFIG_disable_overscan”: “1”,
“RESIN_HOST_CONFIG_disable_splash”: “1”,
“RESIN_HOST_CONFIG_dtoverlay”: “"vc4-fkms-v3d"”,
“RESIN_HOST_CONFIG_dtparam”: “"audio=on","i2c_arm=on","spi=on","audio=on"”,
“RESIN_HOST_CONFIG_enable_uart”: “0”,
“RESIN_HOST_CONFIG_gpu_mem”: “16”,
“RESIN_HOST_FIREWALL_MODE”: “”,
“RESIN_SUPERVISOR_DELTA”: “1”,
“RESIN_SUPERVISOR_POLL_INTERVAL”: “900000”,
“RESIN_SUPERVISOR_DELTA_REQUEST_TIMEOUT”: “59000”
}
}
}

This is without a docker container loaded. The docker container does not have lines to restart.

13

Is anyone else having this issue? If you start a fresh fleet of raspberry pi, does it not have this problem?

If I hadn’t already invested significant time building on balena, I would have already abandoned it if it caused this trouble on instantiating the first fleet.

14

I have tried deleting my balena account entirely and starting a fresh one with a completely different email, and I still have this issue.

Am I the only one experiencing this issue? Why isn’t the forum flooded with requests?

If this was a production fleet, it would be down for nearly 48 hours.

15

24796ed5ad6c5dfa7a3e7a149ceb3a39_diagnostics_2024.07.23_19.20.00+0000.txt (406.5 KB)

here is a new device diagnostics on a new balena account, new fleet, etc.

no software pushed to the device, but still seeing the supervisor restart over and over.

I notice in this diagnostics, my WiFi SSID is no longer visible as it was on the previous diagnostic, so it seems things are changing on the backend.

16

@jason37 did you try in previous versions and you don’t see this behaviour? Could you please confirm the other versions that you tried?

I will try to reproduce during this morning!

17

@jason37 i could reproduce this on a device in my lab!

in the other hand, could you please try balena ps -a and share the logs? From our hypothesis the supervisor should look healthy (as it is from your diagnostics logs).

i will connect with the balena team to explore what happens.

Thanks for reporting!

18

Hi Jason,

We’ve been trying to reproduce the issue and we think that the issue is that the logs you reported initially are wrong and may have generated a confusion.

2024-07-22T09:13:36-05:00 Supervisor starting
2024-07-22T09:13:42-05:00 Applying configuration change {“SUPERVISOR_POLL_INTERVAL”:“900000”,“SUPERVISOR_DELTA”:“1”,“SUPERVISOR_DELTA_VERSION”:“3”}
2024-07-22T09:13:42-05:00 Applied configuration change {“SUPERVISOR_POLL_INTERVAL”:“900000”,“SUPERVISOR_DELTA”:“1”,“SUPERVISOR_DELTA_VERSION”:“3”}
2024-07-22T09:13:44-05:00 Creating network ‘default’
2024-07-22T09:13:44-05:00 Creating volume ‘resin-data’
2024-07-22T09:13:44-05:00 Creating network ‘default’
2024-07-22T09:13:36-05:00 Supervisor starting
2024-07-22T09:13:42-05:00 Applying configuration change {“SUPERVISOR_POLL_INTERVAL”:“900000”,“SUPERVISOR_DELTA”:“1”,“SUPERVISOR_DELTA_VERSION”:“3”}
2024-07-22T09:13:42-05:00 Applied configuration change {“SUPERVISOR_POLL_INTERVAL”:“900000”,“SUPERVISOR_DELTA”:“1”,“SUPERVISOR_DELTA_VERSION”:“3”}

If you look at the timestamp you’ll see that it’s the same set of logs that are appearing.
So it’s not the supervisor restarting, but the same logs. If you minimize the window and open it again, you will only see one correct set. We’ve already reported this internally.

You can verify that the supervisor is not restarting by executing on the remote shell balena ps -a and checking the time the container named balena-supervisor has been running.

Then, there’s this other set of logs you’ve shared:

2024-07-22T10:48:11-05:00 main This is a test
2024-07-22T10:48:12-05:00 Service exited ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:49:14-05:00 Restarting service ‘main sha256:4928fb6e8ef9b7eeb6874560657c1b2bb864ab7f0667d0559425a1b067ccede5’
2024-07-22T10:49:14-05:00 main This is a test
2024-07-22T10:49:16-05:00 Service exited ‘main

What we are seeing here seems to be a normal behavior. Your service exits (the script ends) and the supervisor will restart it. If you want to change this behavior you can define the restart policy in the docker-compose file. This parameter defaults to always and you can read about it here for example.

Let us know if this solves your issue to confirm there’s no major outage of Balena.

1 Like