Our devices include a Wireguard VPN configuration. In previous BalenaOS releases (e.g. 2.50.1), we’ve used a kernel module container to build the necessary module using the out-of-tree wireguard-linux-compat repository and insert it, which has worked great.
However, new OS releases for the Intel NUC include kernel versions > 5.10 – and since 5.6, Wireguard has been available in-tree. Great news for Wireguard – but unfortunately, it also means we can no longer use wireguard-linux-compat, as the module refuses to build on kernel versions that have in-tree support!
It’s also unfortunately not possible to build the in-tree Wireguard module using the same process, as it depends on some of the crypto APIs that are not included in the host OS kernel. The upshot of all this is that we’re not currently able to upgrade to newer host versions, and I’ve ran out of ideas about how to proceed.
Is there any likelihood that we might see the Wireguard module available in the standard host OS build sometime soon? Failing that, are there any other ideas in the community about how we might be able to proceed?
Hi @matthewmacleod
Thanks for reaching out. I am not super familiar with some of the kernel module building stuff, but would try to clarify some of the stuff here. So as to one of our colleague from OS side can help more.
As I understand you need the wireguard-linux-compat for compiling and building some of the dependent modules. I see a thread for debian which mentions about wireguard-tools build with 5.10 and above. The user was able to run the necessary module build with Wireguard available in-tree. Ref: Moving Linux Kernel to 5.10 - Page 21 - Raspberry Pi Forums (a debian ref. but does mainly point to Wireguard dependency).
If wireguard-tools is not what you are after and looking for something else. Please do let us know, in the mean time I’ll tag our OS folks internally to take a look at the thread.
For some background, the wireguard-linux-compat repository is the official source of the Wireguard kernel module for kernel versions < 5.6. In kernel version 5.6, this was merged into the mainline – so there was no longer a need for users to use it to get access to Wireguard. The old repository will detect the kernel version it is building against and refuse to build.
With older OS releases, we used this repository to build the Wireguard module and everything worked great. But the latest release of BalenaOS that I’ve upgraded to now includes kernel 5.10, which means this repository refuses to build. I’d switch to using the in-tree module, but I haven’t been able to get it to build against the kernel source for that latest BalenaOS either – I think because the kernel supplied with the OS doesn’t have all of the necessary built-in configuration options turned on.
What would be great would be if it were possible to make Wireguard more generally available by including either the module itself—or at least the kernel API dependencies—in the supplied host OS kernel build. This will hopefully be something the OS team can provide some guidance on!
In the meantime—hopefully helpful to anyone else who comes across this thread—I was able to resolve the problem by patching the wireguard-linux-compat module to build against newer kernels. This means I can keep using the out-of-tree module for the moment. This isn’t a long-term solution, because at some point it will not be possible to build the module against newer kernels. But as a stop-gap solution it’s worked out.
Hey @matthewmacleod, would you mind sharing the patch you made to wireguard-linux-compat to enable newer kernels? I understand it’s a temporary workaround but it may help other users in the same situation, like myself!
No problem @klutchell – it’s an extremely simple patch to remove the version check, and after this the module builds and works as expected (at least on 2.83.1+rev8 for the Intel NUC)
As for balenaOS, we would like to enable the wireguard module by default in the kernel if there is demand for it, but we have to consider the rootfs space concerns for some device types, and devices running older kernels obviously would be excluded.
I noticed you are using an Intel NUC. We do have a similar OS for Generic x86_64 that typically gets a much wider net of kernel modules and firmware and still runs on the NUC. For now we still haven’t enabled wireguard on any devices, but would it be helpful if we tried enabling it on the Generic x86_64 device in the short term?
Note that this Generic device type is also under some flux right now so if your current workaround is enough it might be best to wait until we migrate Generic x86_64 to a new repo before we start adding kernel features.
Thanks @klutchell – hadn’t considered looking at generic-x86-x64!
A migration is probably out-of-scope for the moment, as we’ve got some devices out there that wouldn’t be an easy upgrade, so I’d need to keep everything working on the NUC OS in any case. The workaround is suitable for now, but I’ll keep my eye on developments around the generic device in case we get the opportunity. I do reckon Wireguard will be quite popular given the ease of setup, but totally appreciate the constraints involved.
But +1 if we could replace OpenVPN with WireGuard …
Probably a bit more work, but could it be an option to either have one or the other as different BalenaOS builds to manage local resources?
I am currently running WireGuard with my devices and evaluating migrating to OpenBalena / BalenaCloud … Got my first device up and running today in BalenaCloud using most of my application stack (without WireGuard) …
Have you tried to run your Wireguard server as a client in a multicontainer environment?
Not sure what I am doing wrong …
I grabbed a copy of your github code, placed it in a directory, moved the content from the docker-compose file into the parent one I have and removed details about the server settings - expecting it to start and wait for a wg0 config file …
However, I am getting the error on build of:
Uploading source package to balenaClouddata/services/wireguard should NOT have additional properties
The respective part of my docker-compose is:
wireguard:
build: ./balena-wireguard
privileged: true
volumes:
- wireguard-config:/etc/wireguard
sysctls:
- net.ipv4.conf.all.src_valid_mark=1
labels:
- io.balena.features.kernel-modules=1
This should work in your setup too even if you only copied the wireguard service. Is your full docker-compose file available to review on GitHub? It got a bit mangled when pasting here so I can’t easily check it for errors.
Uploading source package to balenaClouddata/services/wireguard should NOT have additional properties
That kind of error almost always means a syntax error or typo in the compose file, or perhaps an unsupported field but these all work for me as per our docs: docker-compose.yml fields - Balena Documentation
Hey @malharwood, I found some configuration bugs in my latest wireguard image and I’m working on a rework in my free time. Hopefully I’ll have a stable version soon, but I’d love to hear how it worked out for you and what changes you had to make!
I just forked your github and added my changes into a new ‘DedicatedClient’ branch.
Changes:
moved all files into a subdirectory
created multi container docker-compose.yml
– created basic apache container as end node
– created new ‘wireguard’ container referring to your build in the subfolder
– modified the peer.conf file to allow full configuration
– modified run.sh considerably to remove server config and just be a client (left yours commented)
** note this is a copy of my current one as I have a couple of other containers in there, but think I brought everything across ok.
At the device variables level I added the 8 variables (or obviously balance between fleet variables for most of them and just specific ones for each device)
@klutchell - one thing I did notice with this build that I cant seem to work out!
In a previous configuration, I somehow had it so that the BalenaCloud device summary screen was showing the local IP Address for the physical interface as well as the WireGuard interface …
But it is not now
I am not sure how I did it before, and hence cant repeat it …
I somehow had it so that the BalenaCloud device summary screen was showing the local IP Address for the physical interface as well as the WireGuard interface
@malharwood You know what? I’ve seen the same thing where sometimes the wg interface is shown and sometimes it is not, though the behaviour doesn’t change otherwise. I haven’t figured this out yet either!
Would you be willing to share your repository URL so we can learn what you’ve done and why?
