Hello,
I’ve been diagnosing an issue on multiple devices in our balena fleet that have been stuck in the Online (Heartbeat only) status after flashing them with balenaOS version 2.95.12+rev1. They have been stuck in this state for a day now. The issue didn’t appear related to network connectivity as the logs from the devices were being streamed to balenaCloud. I also went through the steps for diagnosing firewall issues but there were no issues there.
After checking multiple services, I narrowed the issue down to the os-config service which was stuck in a crash loop:
root@48998eb:~# journalctl --follow -n 300 -u os-config
Feb 09 10:12:07 48998eb os-config[465032]: Fetching service configuration from https://api.balena-cloud.com/os/v1/config...
Feb 09 10:12:07 48998eb systemd[1]: os-config.service: Main process exited, code=killed, status=11/SEGV
Feb 09 10:12:07 48998eb systemd[1]: os-config.service: Failed with result 'signal'.
Feb 09 10:12:18 48998eb os-config[465150]: Fetching service configuration from https://api.balena-cloud.com/os/v1/config...
Feb 09 10:12:18 48998eb systemd[1]: os-config.service: Main process exited, code=killed, status=11/SEGV
Feb 09 10:12:18 48998eb systemd[1]: os-config.service: Failed with result 'signal'.
Feb 09 10:12:28 48998eb os-config[465270]: Fetching service configuration from https://api.balena-cloud.com/os/v1/config...
Feb 09 10:12:28 48998eb systemd[1]: os-config.service: Main process exited, code=killed, status=11/SEGV
Feb 09 10:12:28 48998eb systemd[1]: os-config.service: Failed with result 'signal'.
Feb 09 10:12:38 48998eb os-config[465314]: Fetching service configuration from https://api.balena-cloud.com/os/v1/config...
Feb 09 10:12:38 48998eb systemd[1]: os-config.service: Main process exited, code=killed, status=11/SEGV
Feb 09 10:12:38 48998eb systemd[1]: os-config.service: Failed with result 'signal'.
When I tried to manually run the os-config update command I got the same error (i.e., segmentation fault):
root@48998eb:~# os-config update
Fetching service configuration from https://api.balena-cloud.com/os/v1/config...
Segmentation fault
I reported this issue on the Github repository of os-config but haven’t gotten a response yet. Any help in resolving this issue is appreciated.
System information
root@48998eb:~# os-config --version
os-config 1.2.1
root@48998eb:~# cat /etc/os-release
ID="balena-os"
NAME="balenaOS"
VERSION="2.95.12+rev1"
VERSION_ID="2.95.12+rev1"
PRETTY_NAME="balenaOS 2.95.12+rev1"
MACHINE="genericx86-64-ext"
META_BALENA_VERSION="2.95.12"
BALENA_BOARD_REV="ed67aa9"
META_BALENA_REV="87974875"
SLUG="genericx86-64-ext"