Setting kernel parameters in BalenaOS

My current production application stack runs on Ubuntu 18.04 on a intel device. The device is very busy processing live video so to get back some cycles I disabled all mitigations due to the various Intel processor bugs, like so:

In /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT=“noresume noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off mitigations=off”

This decreased CPU loading almost 24% for my use-case and made the device performant enough for my payload.

Where does one set the same parameters for the BalenaOS kernel ?

It’s unfortunately not possible to set kernel parameters on balenaOS, unless you make a custom build of the OS.

That said, it’s something we consider desirable and hope to add at some future version of the OS, but there’s no ETA unfortunately.

That’s bad news. I’m willing to do the custom build work, are there any caveats in terms of BalenaCloud integration or the like that I need to be aware of before hand? I do not want to spend a lot of time doing this and then learning about some services that are not available fro custom builds, for example.

Also some from the hip-pointers on going about building a custom BalenaOS would be appreciated. I’m generally Linux savvy so getting to grips with the tool chain does not concern me much, I think…

This should get you started: https://www.balena.io/os/docs/custom-build/

A limitation of custom builds is that they don’t currently support remote host OS updates via the balenaCloud infrastructure.

I discussed this a bit with the OS team and suggested that you could do it on a device by device basis by entering the GRUB menu and editing the kernel boot parameters. It’s not ideal if the fleet is large, and you’d have to go through the same process after each host OS update, but given that the parameters are not critical for the board to boot, we don’t see any other immediate downsides to it and should get you going.

Hi, just to qualify the above, editing the grub menu will allow you to test the command line changes which you can then integrate in a custom build as discussed previously.

Thanks - to clarify, if I deploy a custom build, do I have to recall all devices for a manual reflash once the functionality is available in the stock OS, or can I replace the custom OS with the stock OS ‘over the air’ ?

It will be possible to do the updates over the air, but will require some work for your side as it won’t be automatically available in our dashboard, because we wouldn’t have reference to where your new OS versions are or named. We are also working on some ways to improve this flow so that customers can add and maintain their own device types and have them show up in the cloud, but that will probably only be released end of the year or 2021 I would estimate.

Is there no way to get something like the below from /mnt/boot/EFI/BOOT/grub.cfg to work ?
The ‘grub compile’ tools I usually use does not seem to be present in the root OS, but that may be called something else.

 ....
 grub_cmdline="${grub_cmdline} uuid_rootb=${uuid_rootb}"
 grub_cmdline="${grub_cmdline} uuid_state=${uuid_state}"
 grub_cmdline="${grub_cmdline} uuid_data=${uuid_data}"
fi

grub_cmdline="${grub_cmdline} noresume noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off mitigations=off"

menuentry 'boot'{
if [ ${bootcount} = 2 ] ; then
 if [ ${resin_root_part} = 2 ] ; then
  search --set=root --label resin-rootB
  linux /boot/bzImage root=LABEL=resin-rootB rootwait intel_idle.max_cstate=1 ${grub_cmdline}
 else
....

Hi @sthysel, I had a chat with Alex about his and he explains that user space grub tools like grub-install can’t be used in this case. Even if they were installed in balenaOS, the root filesystem is read-only. The OS is not designed to work that way. The grub installation is done by the Yocto build process, so the changes in grub.cfg need to be done in a custom build.

So please take a look on the linked docs how to compile a custom build. As my teammates explained the main issue with custom builds currently is the lack of support on self-serve balenaCloud host OS updates.

We raised a feature request on your behalf in our internal system. We’ll ping once balenaCloud supports your use case. Meanwhile, let us know how it goes, if you go forward with the custom build route.

Hi @sthysel,

grub.cfg could be in two possible locations depending on your NUC.
In /mnt/boot/grub/grub.cfg or /mnt/boot/EFI/BOOT/grub.cfg. This varies from NUC to NUC. Some use EFI. Some use classic. balenaOS for the nuc ships with both and the choice is made at runtime while flashing.

You can ssh into the Host OS and manually edit the kernel cmdline in grub.cfg.

I think the key bit you are missing is this:
If you are used to the Ubuntu way, you will be thinking of having a step like grub install.
In balenaOS, grub.cfg is live. Changes to /mnt/boot/xxx/grub.cfg will happen on next boot.

All the lines starting with linux /boot/bzImage will need your extra kernel cmdline.

Updating the hostOS from the dashboard will overwrite your grub.cfg changes with the default one that ships with balenaOS.

If the fleet size is small, I’d suggest manually doing this. If the fleet size is large, you could look at writing a script that does this change

Keep in mind, a bad change to grub.cfg can easily brick a device. And power failure during the change can also brick the device.

Regards
ZubairLK