Hi all. I don’t have a lot of info to share because this just happened. We have some x86 devices that got updated to OS v8.0.0. I ran into an issue where one of my containers was not starting because I have a gate that checks for all the envs being set, which they were, so it should have started. Attempting to update to v8.0.0+rev2 has caused the device to drop offline and not recover. I don’t have anyone at the site so kind of stuck right now.
Edit the env problem is a red herring, caused by something else.
Thank you for reporting this. Could you share more specific information about the device type and hardware you are using? I understand you observed an issue going from v8.0.0 to v8.0.0+rev2? What was the OS version before updating to v8?
I have to preface this by saying this may have been a coincidental hardware failure, but the timing is certainly odd. We’ve seen this happen with a couple other devices in the past and I cannot be sure if the v8 update had anything to do with it. I will test this update path with local devices. I have one other device in the field with v8.0.0 right now and I’m concerned about updating it until I know more.
As for further info. These are small industrial PCs, I will leave the manufacturer name out of it. They run an Intel Celeron J3455, dual NIC, small nvme drive. The device would have been running OS 7.x before. Someone very eager on my team would have updated it to v8.0.0 in the last few days. I thought it was having an issue with the envs as I described so I attempted to update to 8.0.0+rev2. It never came back after reboot. We had someone go on site and it seems the device no longer boots past the post screen. I can’t know if this is pure coincidence or something the OS update did.
Update on this. I have run this test on local devices. It seems to work just fine. I think it must have been a coincidence and the device died after the reboot instead of the update.
Another update. I just attempted to update all my devices in my fleet to 7.7.0+rev2. A number of devices have experienced the exact same failure. Out of total of six, three were still on 6.11.10. Not sure about the others, it’s not showing me anymore. They never came back after the reboot. I am requesting that the original device is shipped to me so I can investigate whether this is software or hardware. These are all Generic x86_64 (GPT) devices.
Hi @pinter, I just successfully upgraded a similar spec device from balenaOS 6.11.10 to balenaOS 7.7.0+rev2. It worked as expected. Which matches your observations on local devices. Can you think of anything that is out of the ordinary on the field devices that failed? Using secure boot? Different connectivity on the field devices e.g. different wifi adapters or different modems?
Smart of you to get a failed device back. If you are able to access the drive, you can check the HUP logs in /mnt/data/balenahup.
Hi Ross, thanks for the reply. I should specify these devices are using full disk encryption as well. They all start life the same way; our panel builder uses an image of OS 6.5.x which, at the time, required the custom config to enable FDE. After that they usually get upgraded, when they land on site, to a newer version. Up to recently they would get upgraded to whatever the latest OS was but I’ve instructed to not do that and we’re trying to only go as far as 7.7.0rev+2. Right now those failures are very inconsistent.
I should have one of the failed devices soon so I will get a better idea of what we’re dealing with. My plan for the device is to check a few things first
Boot the device, see if it is still stuck.
Boot the device and attempt to access the BIOS, if I can’t do that then it’s pretty clear where the problem lies.
If I can access the BIOS I want to check a few settings and maybe “Reset to defaults”, see if it will boot.
If I get this far then I will attempt to reload Balena again, see if it will boot.
If I get through all the steps, including reloading the OS, and it boots then I’m really confused, and concerned about the rest of my fleet. I suppose that if I find a hardware issue I’m equally concerned.
Let me know if there’s anything specific you’d like me to check on that device. This device is the original one which was on OS v8 and failed going to v8.0.2, but it’s the same symptoms. I appreciate you testing this out on your end. I know that the best thing would be to reproduce the issue so I will do what I can over here.
Question for you. The implementation of FDE you guys created is FDE and secure boot. I think the old implementation that I enabled via the config file was just FDE, without secure boot. Can I still only enable FDE, without secure boot, in newer versions of the OS? As I mentioned, all our boxes are provisioned with that old 6.5 image. Can I achieve the same with a newer image, say 7.7.0?
8GB should be enough. But felt I should ask given the age of the devices.
balena’s implementation has always been secure boot + full disk encryption (SB + FDE). That is, we implemented them together and you can’t have one without the other. A while back we made it a bit easier by putting a SB + FDE toggle in the dashboard UI so you don’t have to edit a config file or use the CLI.
BTW, when you do your testing, I suggest using Ethernet - eliminates wifi as a variable. wifi is super useful of course, but I find it gets in the way of diagnosing.
Hello. I got the device today and have been doing some tests.
First off, I don’t think this is a hardware failure. I can access the BIOS, I took the SSD out and it boots straight into BIOS, I eventually booted off a live usb and everything is fine. Basically nothing I tried indicated to me the device died.
Further testing involved booting off the live usb. I used Alpine linux for this. Caveat to mention right off the top is that in order to boot off the USB I had to go into the BIOS and set “Attempt Secure Boot” to “Disabled”. I don’t think this wipes the keys on this device.
I used Claude to help me with some of these diags. Booted the device
/dev/sda appears intact
blkid shows device has it’s boot plus LUKS partitions
Mounted the boot partition to have a look
grubenv showed
upgrade_available=1
bootcount=2
resin_root_part=3
The LUKS headers look healthy. Claude noted that the Tokens section was empty and to flag this.
bzImage, grub.cfg, grub.cfg.sig, grubenv are all present
I tried to reset the grubenv values and even change resin_root_part with no luck. I set the Attempt Secure Boot back to Enabled each time I made any change to grubenv (although setting it to Disabled in the first place may have voided everything anyway). I used the proper utility to set the grubenvs. Unfortunately the grub.cfg is signature locked so I cannot edit it to output any useful data. I’m leaving the device in this state for now, if there is some way we could make changes to grub.cfg, or if you have some voodoo, to get output, that would be ideal. At least maybe we could see what’s panicking.
I guess a lot of this points to some kind of TPM disturbance or something to that effect, I can’t think of what it could be however. Something that disturbed the PCRs? What I do know is that there is absolutely no way I could do any of this kind of troubleshooting remotely and I have very real concerns about doing any more updates on the remaining devices atm.
Need some capability to dump logs in these kinds of situations.
One more thought I had here. What influence, if any, does supervisor have on this whole upgrade process? The devices that were on 6.11, I’m almost certain they would have had a pretty old supervisor and I think the OS update was trying to update it as well.