My current production application stack runs on Ubuntu 18.04 on a intel device. The device is very busy processing live video so to get back some cycles I disabled all mitigations due to the various Intel processor bugs, like so:
That’s bad news. I’m willing to do the custom build work, are there any caveats in terms of BalenaCloud integration or the like that I need to be aware of before hand? I do not want to spend a lot of time doing this and then learning about some services that are not available fro custom builds, for example.
Also some from the hip-pointers on going about building a custom BalenaOS would be appreciated. I’m generally Linux savvy so getting to grips with the tool chain does not concern me much, I think…
I discussed this a bit with the OS team and suggested that you could do it on a device by device basis by entering the GRUB menu and editing the kernel boot parameters. It’s not ideal if the fleet is large, and you’d have to go through the same process after each host OS update, but given that the parameters are not critical for the board to boot, we don’t see any other immediate downsides to it and should get you going.
Hi, just to qualify the above, editing the grub menu will allow you to test the command line changes which you can then integrate in a custom build as discussed previously.
Thanks - to clarify, if I deploy a custom build, do I have to recall all devices for a manual reflash once the functionality is available in the stock OS, or can I replace the custom OS with the stock OS ‘over the air’ ?
It will be possible to do the updates over the air, but will require some work for your side as it won’t be automatically available in our dashboard, because we wouldn’t have reference to where your new OS versions are or named. We are also working on some ways to improve this flow so that customers can add and maintain their own device types and have them show up in the cloud, but that will probably only be released end of the year or 2021 I would estimate.
Is there no way to get something like the below from /mnt/boot/EFI/BOOT/grub.cfg to work ?
The ‘grub compile’ tools I usually use does not seem to be present in the root OS, but that may be called something else.
....
grub_cmdline="${grub_cmdline} uuid_rootb=${uuid_rootb}"
grub_cmdline="${grub_cmdline} uuid_state=${uuid_state}"
grub_cmdline="${grub_cmdline} uuid_data=${uuid_data}"
fi
grub_cmdline="${grub_cmdline} noresume noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off mitigations=off"
menuentry 'boot'{
if [ ${bootcount} = 2 ] ; then
if [ ${resin_root_part} = 2 ] ; then
search --set=root --label resin-rootB
linux /boot/bzImage root=LABEL=resin-rootB rootwait intel_idle.max_cstate=1 ${grub_cmdline}
else
....
Hi @sthysel, I had a chat with Alex about his and he explains that user space grub tools like grub-install can’t be used in this case. Even if they were installed in balenaOS, the root filesystem is read-only. The OS is not designed to work that way. The grub installation is done by the Yocto build process, so the changes in grub.cfg need to be done in a custom build.
So please take a look on the linked docs how to compile a custom build. As my teammates explained the main issue with custom builds currently is the lack of support on self-serve balenaCloud host OS updates.
We raised a feature request on your behalf in our internal system. We’ll ping once balenaCloud supports your use case. Meanwhile, let us know how it goes, if you go forward with the custom build route.
grub.cfg could be in two possible locations depending on your NUC.
In /mnt/boot/grub/grub.cfg or /mnt/boot/EFI/BOOT/grub.cfg. This varies from NUC to NUC. Some use EFI. Some use classic. balenaOS for the nuc ships with both and the choice is made at runtime while flashing.
You can ssh into the Host OS and manually edit the kernel cmdline in grub.cfg.
I think the key bit you are missing is this: If you are used to the Ubuntu way, you will be thinking of having a step like grub install. In balenaOS, grub.cfg is live. Changes to /mnt/boot/xxx/grub.cfg will happen on next boot.
All the lines starting with linux /boot/bzImage will need your extra kernel cmdline.
Updating the hostOS from the dashboard will overwrite your grub.cfg changes with the default one that ships with balenaOS.
If the fleet size is small, I’d suggest manually doing this. If the fleet size is large, you could look at writing a script that does this change
Keep in mind, a bad change to grub.cfg can easily brick a device. And power failure during the change can also brick the device.
In the latest BalenaOS I noticed a new file /mnt/boot/EFI/BOOT/grub_extraenv (it may have been there before), which gets sourced in grub.cfg. So I added the mitigation switch available in kernels 5.2 and up like below:
extra_os_cmdline="mitigations=off"
Can team balena confirm that /mnt/boot/EFI/BOOT/grub_extraenv will stay put between BalenaOS upgrades ?
I’ll ask the OS team to confirm, but I’m fairly certain we’re not prepared to make any guarantees of that sort for the base OS. How did you end up adding the boot flags? Did you create a custom build or manually patch each device?
Manually patch each device. The parameters need to be set to get the performance I require from the device. I of course would much rather use dashboard managed variables, much in the same way environment variables can be set. There does not seem to be a way to do that unfortunately, or is there ?
Hi @sthysel , we have created internal issue for this. I noted that it’s preferable to be set from the dashboard. We’ll get back to you once the functionality is implemented.
Hi, we have recorded the feature request to be able to modify kernel command line arguments. There are some valid cases for this, but unfortunately the feature request has not yet added enough weight to be considered by the product team.