Supervisor restart loop after OS update

jpmeijers · July 11, 2023, 3:16pm

For the first time in likely more than a year I opened the balena dashboard to reconfigure a remote device. The console popped up with a lot of warning about upgrading otherwise my device will go offline after 1 August.

This was the first I hear about this. I would expect an email warning me that my devices will go offline.

I then tried updating the supervisor and host OS, which failed most of the time. In the end after many tries I managed to get my device on OS v3.0.5. Trying to update to anything later fails with the following warning in the logfile under /mnt/data/balenahup

root@b0b34a0:~# cat /mnt/data/balenahup/upgrade-2.x.sh.20230711_151019.log 
================upgrade-2.x.sh HEADER START====================
Tue Jul 11 15:10:19 UTC 2023
[000000000][LOG]Raw target version: 3.0.6
[000000001][LOG]Loading info from config.json
[000000003][LOG]Target version supports hostapps, no device type support check required.
[000000003][LOG]Target OS version "3.0.6" OK.
[000000003][LOG]OS variant: 3.0.5
[000000003][ERROR]Host OS version "3.0.5" not supported.

root@b0b34a0:~# cat /mnt/data/balenahup/upgrade-2.x.sh.20230711_150734.log 
================upgrade-2.x.sh HEADER START====================
Tue Jul 11 15:07:34 UTC 2023
[000000000][LOG]Raw target version: 3.0.8
[000000002][LOG]Loading info from config.json
[000000003][LOG]Target version supports hostapps, no device type support check required.
[000000003][LOG]Target OS version "3.0.8" OK.
[000000003][LOG]OS variant: 3.0.5
[000000003][ERROR]Host OS version "3.0.5" not supported.

In the meantime the log output on the dashboard is spammed constantly with:

Applying boot config: {}
Applied boot config: {}
Applying boot config: {}
Applied boot config: {}

And looking at the supervisor container’s output I see:

[error]   Expected splash image to be a base64 encoded PNG image. Skipping write.
[event]   Event: Apply boot config success {}
[event]   Event: Apply boot config in progress {}
[event]   Event: Apply boot config success {}
[event]   Event: Apply boot config in progress {}
[error]   Expected splash image to be a base64 encoded PNG image. Skipping write.
[event]   Event: Apply boot config success {}
[event]   Event: Apply boot config in progress {}
[event]   Event: Apply boot config success {}
[event]   Event: Apply boot config in progress {}

jpmeijers · July 11, 2023, 3:33pm

In the current state the dashboard is stuck saying “updating” and “Downloading” for the service. But the service never updates, while the old one is running.

jpmeijers · July 13, 2023, 7:07am

After about 12 hours this device went offline. It’s now a day later and its still offline. So the required updated resulted in me losing this device

jpmeijers · July 26, 2023, 12:45pm

Today my device came back online. It still showed “updating” and the service wasn’t running. I was however able to do an update to OS v3.1.0 and that also updated the supervisor to v14.11.12.

After the OS update, and another 10 minute wait for the device to come back online, it still says “updating” even though we’re on the latest OS and supervisor. The service is still not running.

I now went to GitHub - xoseperez/basicstation: Basics™ Station Packet Forward protocol using Docker and clicked on the install button and chose my existing fleet. Still stuck on “updating” and the service is not running.

Support access has been granted if anyone wants to have a look.

jpmeijers · July 26, 2023, 12:46pm

Also still looping the “applying device config”

root@b0b34a0:~# journalctl -xe
Jul 26 12:45:59 b0b34a0 566322f82c45[1978]: [error]   Expected splash image to be a base64 encoded PNG image. Skipping write.
Jul 26 12:45:59 b0b34a0 balena-supervisor[2361]: [error]   Expected splash image to be a base64 encoded PNG image. Skipping write.
Jul 26 12:45:59 b0b34a0 566322f82c45[1978]: [event]   Event: Apply boot config success {}
Jul 26 12:45:59 b0b34a0 balena-supervisor[2361]: [event]   Event: Apply boot config success {}
Jul 26 12:45:59 b0b34a0 566322f82c45[1978]: [warn]    Ignoring unsupported or unknown compose fields: containerName
Jul 26 12:45:59 b0b34a0 balena-supervisor[2361]: [warn]    Ignoring unsupported or unknown compose fields: containerName
Jul 26 12:45:59 b0b34a0 566322f82c45[1978]: [event]   Event: Apply boot config in progress {}
Jul 26 12:45:59 b0b34a0 balena-supervisor[2361]: [event]   Event: Apply boot config in progress {}
Jul 26 12:45:59 b0b34a0 566322f82c45[1978]: [event]   Event: Apply boot config success {}
Jul 26 12:45:59 b0b34a0 balena-supervisor[2361]: [event]   Event: Apply boot config success {}
Jul 26 12:45:59 b0b34a0 566322f82c45[1978]: [event]   Event: Apply boot config in progress {}
Jul 26 12:45:59 b0b34a0 balena-supervisor[2361]: [event]   Event: Apply boot config in progress {}
Jul 26 12:45:59 b0b34a0 balena-supervisor[2361]: [error]   Expected splash image to be a base64 encoded PNG image. Skipping write.
Jul 26 12:45:59 b0b34a0 566322f82c45[1978]: [error]   Expected splash image to be a base64 encoded PNG image. Skipping write.
Jul 26 12:45:59 b0b34a0 balena-supervisor[2361]: [event]   Event: Apply boot config success {}
Jul 26 12:45:59 b0b34a0 566322f82c45[1978]: [event]   Event: Apply boot config success {}
Jul 26 12:46:00 b0b34a0 balena-supervisor[2361]: [warn]    Ignoring unsupported or unknown compose fields: containerName
Jul 26 12:46:00 b0b34a0 566322f82c45[1978]: [warn]    Ignoring unsupported or unknown compose fields: containerName
Jul 26 12:46:00 b0b34a0 566322f82c45[1978]: [event]   Event: Apply boot config in progress {}
Jul 26 12:46:00 b0b34a0 balena-supervisor[2361]: [event]   Event: Apply boot config in progress {}
Jul 26 12:46:00 b0b34a0 balena-supervisor[2361]: [event]   Event: Apply boot config success {}
Jul 26 12:46:00 b0b34a0 566322f82c45[1978]: [event]   Event: Apply boot config success {}
Jul 26 12:46:00 b0b34a0 566322f82c45[1978]: [event]   Event: Apply boot config in progress {}
Jul 26 12:46:00 b0b34a0 balena-supervisor[2361]: [event]   Event: Apply boot config in progress {}
Jul 26 12:46:00 b0b34a0 566322f82c45[1978]: [error]   Expected splash image to be a base64 encoded PNG image. Skipping write.
Jul 26 12:46:00 b0b34a0 balena-supervisor[2361]: [error]   Expected splash image to be a base64 encoded PNG image. Skipping write.
Jul 26 12:46:00 b0b34a0 566322f82c45[1978]: [event]   Event: Apply boot config success {}
Jul 26 12:46:00 b0b34a0 balena-supervisor[2361]: [event]   Event: Apply boot config success {}
Jul 26 12:46:00 b0b34a0 balena-supervisor[2361]: [warn]    Ignoring unsupported or unknown compose fields: containerName
Jul 26 12:46:00 b0b34a0 566322f82c45[1978]: [warn]    Ignoring unsupported or unknown compose fields: containerName
Jul 26 12:46:00 b0b34a0 566322f82c45[1978]: [event]   Event: Apply boot config in progress {}
Jul 26 12:46:00 b0b34a0 balena-supervisor[2361]: [event]   Event: Apply boot config in progress {}

jpmeijers · July 26, 2023, 12:49pm

Forcing the service to run pops up with this message:

Error!Request error: App not found: an app needs to be installed for this endpoint to work. If you’ve recently moved this device from another app, please push an app and wait for it to be installed first.

jpmeijers · July 26, 2023, 1:06pm

It seems like the default no splash image causes the supervisor to go into a restart loop. I defined an override splash image now, and that seems to make the supervisor run and download the service.

alexgg · August 3, 2023, 6:22pm

This is a known issue `Applying boot config {}` loop without reboot when `balena-logo-default.png` is size 0 · Issue #1982 · balena-os/balena-supervisor · GitHub. Could you please self-serve an update of the supervisor?

jpmeijers · August 4, 2023, 1:11pm

As you can see I am running supervisor v14.11.12, which is already the latest.