I’ve just been notified by GitHub that version 3 is launched with support for newer Balena CLI versions and newer balenaOS versions. Awesome work! I’ll try it out as soon as possible.
I just have a couple of questions about the upgrade.
Is it as simple as pulling openBalena again and use ./scripts/compose up -d? In other words, just upgrading the versions of the containers? (I’m asking this because of the Kubernetes support I’ve built).
I know you’ve to update your devices to BalenaOS v2.58.3 or higher in order for it to work properly. I also know openBalena doesn’t provide in HostOS updates (v3 still doesn’t support this, right?). Can I manually update every device by hand using the VPN tunnel? And should I update the devices first or openBalena?
The changelog of openBalena, should I just take a look at every container’s changelog in order to know what’s new? It’s not a problem at all, but maybe someone can highlight the new (awesome) features in V3 of openBalena.
I’ve said it before, but awesome work! openBalena improved our fleet management, update flow and development flow and we’re still very excited about using openBalena and contributing where we can!
Hello, thanks for the kind words. To answer your questions:
Yes, it’s as simple as stopping the services, pulling the repository and starting the services again.
Devices currently provisioned with older than required versions will keep running their applications but certain functionality is incompatible with the new openBalena release. Those devices will need to be reprovisioned to the newer balenaOS versions. The order doesn’t matter and there is going to be a short period of time the devices will log errors either way.
I scanned through the changelogs of the various components to come up with proper release notes, but honestly it was a mammoth task for little benefit – the changes are mostly API-related or internal improvements that really matter little to users. I think the CLI and SDK changelogs are far more interesting.
We’re still not where we’d like to be with regards to automating openBalena releases but we’ll try harder in order to keep pace with developments on the backend.
Regarding question 2, is it not possible at all to update the devices manually via SSH instead of reprovisioning the devices? Our devices have some unique credentials in order to communicate with our servers and some of them are already in the hands of our clients. So taking them back and reprovisioning them is a pain in the ass. Or are there some drastic changes on balenaOS which prevents the OS from updating from 2.48.0 to the latest version?
And It’d be awesome to see more automated releases indeed, but I think it’s fair to say that openBalena has improved quite a lot since the first release. I’ll test out upgrading openBalena and upgrading the devices that belong to openBalena. I’ll definitely post updates here regarding the update process!
There have been changes to the host OS updater in recent versions of balenaOS. You are welcome to try (on a test device), but I’m afraid currently the only supported way is to reprovision the devices.
Just an update for other openBalena users, I’m testing out different scenario’s.
I’ll try to test out every scenario I can think of, but don’t pin me on that :).
Some information about all scenario’s:
I’ll only test with Raspberry Pi 4 devices, because these are the only devices we’re using at the moment.
I’ll not test every Balena function, only the functions that I’m using with Balena, which are the VPN, basic device functions (like balena devices, balena env add, balena env rm etc.), local deployment (balena push), local SSH and over VPN, balena app commands, updates and provisioning of devices.
I’m using a VPS/Kubernetes at DigitalOcean. I’ve heard about some problems with non-KVM VPS’s.
Scenario’s I could think of:
Will the VPN still work on balenaOS devices >= 2.58.0 when connected to openBalena v2.x.x I can confirm that this is the case. I’ve updated one of my devices to v2.58.6+rev1 (dev) with supervisor version 11.14.0, and I can still connect via the VPN tunnel to the device. I updated the Host OS first and then the supervisor.
Will the VPN still work on balenaOS devices < 2.58.0 when connected to openBalena v3.x.x I’ve created a clean install of openBalena v2.x.x and checked the VPN with balenaOS 2.48.0 (supervisor 10.8.0), and it worked, as expected. Next, I’ve updated openBalena to v3.0.1 and the device stayed on balenaOS 2.48.0 (supervisor 10.8.0). It showed as ‘online’ on openBalena and used the VPN to SSH into the device, and it still worked . Next, I’ve updated the Host OS to 2.58.6+rev1 and supervisor to v11.14.0. Everything worked as expected!
Does provisioning a balenaOS device >= 2.58.0 work when using openBalena v3.x.x.? In version 3.1.1 the upgrade bug from v2.x.x to v3.x.x is fixed. So provisioning a device requires v3.1.1. But to confirm, this works!
What to expect when upgrading from openBalena v2.x.x to v3.x.x? First thing I noticed is I had to update the Balena CLI, as mentioned in the README. The balena devices command didn’t work. So it’s something to keep in mind when upgrading, because v2.x.x only works till v12.5.0.
Some of the scenario’s above will probably be already tested by the Balena team, but better safe than sorry, right?
If some of you have tested another scenario or can think of another scenario, please reply!
As a side note, I’ve heard people complain about openBalena v2.x.x not being compatible with balenaOS > 2.49.0 and needing minimum 2.58.0 for openBalena v3.x.x. It’s very unfortunate, because there’s no official support for Host OS updates. But I’d still like to express that, in our case, Balena made our fleet management much better. And creating an open-source version of Balena is just awesome. So everything we get is a nice bonus. Keep in mind that this is just goodwill from the Balena team.
Just another update for people who are interested:
I’ve created a fresh openBalena v2 instance once again. Created some devices with balenaOS 2.48.0, deployed some releases, checked if they updated and they did. Next, I backed up all volumes. Destroyed my VPS, created another VPS, applied the backups of the volumes and started openBalena again with the same config directory. All worked fine again and all devices were back up and running. I would advise to test this before upgrading to openBalena v3, because you never know…
Next, I’ve updated openBalena to v3. Worked like a charm. Created another device with balenaOS 2.58.6 and it provisioned and downloaded the latest release . Deployed another release, and the device updated to that release. Unexpectedly, the devices still running on 2.48.0 also updated to the newest release. That was a pleasant surprise! I’d advice to update the devices to 2.58.6 nonetheless.
Like mentioned before, the 2.48.0 devices still were connected to the VPN, so I still can SSH into them in order to perform some tasks or updates.
I’m almost at the point that I’m going to update our production server to openBalena v3 and updating our devices to 2.58.6. I’ve some more testing to do, but everything looks great!
Hey Bart, thanks for the summary, I’m sure it would be helpful to others searching for the same upgrade path. Regarding what you said about the 2.48.0 devices working, is is possible that only a specific path/function had breaking functionality which might end up in unexpected behavior, so we definitely recommend upgrading to 2.58.6+, even if it works with 2.48.0. Cheers!
I’ve finally upgraded my production server to v3.1.1. All worked fine!
However, I’ve updated my development Pi using the hostapp-update command and updated the supervisor and pushed my containers to the Pi via balena push <local-ip>, and my volume was lost where some credentials were saved. I’m trying to reproduce this on another device, and after the host OS update and supervisor update, I still got my volumes there. So I have no idea yet when this happened.
Maybe somebody from the Balena team can explain this to me when the volume will be removed? This was a volume created via local mode. So it’s not a production device that just downloads the releases.
I’ll try this again with a device that doesn’t use local mode and just downloads the releases. If this happens with devices that are in the field, it’s very problematic.
No, I don’t think so. So let me confirm my understanding – this device has always been in Local Mode, you have been pushing code, then updated the host OS, pushed again and found out the volumes wiped? Did you not change anything in the compose file you are pushing in the meantime? If so, that looks like a bug with balena push and Local Mode.
This device was always in Local Mode. I’ve updated the HostOS (from v2.48.0 to v2.60.1), updated the supervisor (from v10.8.0 to v11.14.0) and started pushing code again. My container booted and it checks for a file in the volume, and it said it didn’t exist where it did before. So I checked the volume in /var/lib/docker/volumes, and it was cleared. The volume data was empty at that time.
I didn’t change anything that changes the volume or the volume’s data.
I see. This sounds like a bug with balena push, the new Supervisor or the Supervisor handling Local Mode after a host OS update. We’ll need to dig into this further. Can you please try one more thing? Ensure the device is in Local Mode and the volume exists, then take down the server so the device can’t reach it and reboot the device. Then check if the volume exists. If it does, also try balena push once more and check again for the volume.
So I’ve stopped the API container, rebooted the device and the volume still exists. Waited some time, tried balena push again with a change, so it forces a container restart. After the boot, the volume still exists. So I can’t reproduce it this way…
Thanks. You said you tried to reproduce on other devices as well, but the volume wasn’t wiped on those, even after the host OS update. This sounds a bit concerning and it’d be great to reproduce. I’ll raise this issue internally and see what we find. Under normal operation though, the only case where volumes (and images) are wiped is when toggling Local Mode off, so your production devices will receive the update fine and will not have their volumes wiped.
. Next, I’ve updated the Host OS to 2.58.6+rev1 and supervisor to v11.14.0. Everything worked as expected!