Is it impossible for now to run Wireguard VPN server container on BalenaOS?

drew · June 16, 2020, 6:00pm

I may be wrong, but it seems that for now it is impossible to run wireguard container provided by linuxserver.io because of some missing linux kernel headers in balenaOS.

I am running balena-pihole project on Raspberry Pi 3 (64bit) whith Adafruit PiTFT 2.8" using balenaOS 2.38.0+rev1 because of this and this issue, which does not allow to show the PADD on the higher BalenaOS versions for some reason.

Recently I have decided to add Wireguard to this setup using the mentioned above container, and found out that the container can not create the network interface saying the following:

16.06.20 20:47:34 (+0300)  wireguard  No kernel headers found in the Ubuntu or Debian repos!! Will try the headers from host (if mapped), may or may not work

and

16.06.20 18:33:14 (+0300)  wireguard  [#] ip link add wg0 type wireguard
16.06.20 18:33:14 (+0300)  wireguard  RTNETLINK answers: Operation not supported
16.06.20 18:33:14 (+0300)  wireguard  Unable to access interface: Protocol not supported
16.06.20 18:33:14 (+0300)  wireguard  [#] ip link delete dev wg0
16.06.20 18:33:14 (+0300)  wireguard  Cannot find device "wg0"

I tried doing it manually with the same results. Fast google-fu, lead me here which hints (but may not be right), that newer kernel (>4.17) should work.

I alse tried the newest atm balenaOS 2.51.1+rev1 (32bit), which has the kernel 4.19.75, but the errors are still the same and the wireguard container still can not create the network interface needed…

Anyway, because PiTFT does not work with Balena versions higher then 2.38.1(the issue, mentioned above is still open) I would not want to use higher version, as I love the look of my PADD…

Therefore do you guys have any ideas how can I install linux kernel headers on my 2.38.1 BalenaOS, needed to successfully run Wireguard VPN? I know, I could just buy another device and use PiVPN project, which now supports Wireguard, but I love the idea of Balena and containerized application setup, so posting this here, hoping for your expertise.

Thanks,

anujdeshpande · June 17, 2020, 9:28am

We have a project template for compiling kernel drivers and loading them from a container - https://github.com/balena-os/kernel-module-build 6. It compiles a simple “hello world” type of module, but you can adapt it for usage with Wireguard.

As for configuring the networking interface, you will probably need to run the container that sets up Wireguard in privileged mode and with network mode set to “host”.

drew · June 17, 2020, 11:45am

Of course I am doing this…

  version: '2.1'

  volumes:
    wireguard_config:
    wireguard_lib_modules:

  services:
    wireguard:
      image: linuxserver/wireguard
      container_name: wireguard
      cap_add:
      - 'NET_ADMIN'
      - 'SYS_MODULE'
      environment:
        - 'PUID=1000'
        - 'PGID=1000'
        - 'TZ=UTC'
        - 'SERVERURL=wireguard.domain.com' #optional
        - 'SERVERPORT=51820' #optional
        - 'PEERS=1' #optional
        - 'PEERDNS=auto' #optional
        - 'INTERNAL_SUBNET=10.0.2.0' #optional
      volumes:
        - 'wireguard_config:/config'
        - 'wireguard_lib_modules:/lib/modules'
      ports:
        - '51820:51820/udp'
      sysctls:
        - 'net.ipv4.conf.all.src_valid_mark=1'
      restart: unless-stopped
      network_mode: host

but changing cap_add to priviledged: true changed nothing…
Obviously the problem is with the lack of needed linux kernel headers, which I have no clue how to add manually to my BalenaOS… Is there any detailed info about this (Although, I believe it is not supposed to be that difficult…) Any ideas why them should not be included in the BalenaOS by default?

drew · June 17, 2020, 11:58am

I mean seriously? No one runs wireguard on balena?

Found Wireguard client in container?, which basically contains the answer above (obviously copied…) So I guess the right (and the most efficient) way to run Wireguard on raspberry pi is using PiVPN project… That’s unfortunate…

richbayliss · June 17, 2020, 12:54pm

needed linux kernel headers

The build.sh script in the repo linked above has a step where is pulls in the kernel headers for the given balenaOS version. It uses the AWS CLI to interrogate the bucket and then pulls down the archive. You should be able to do the same, or reverse it to get a link to the archive manually. You should then make a multistage build Dockerfile to first pull the headers and build the module, then copy from that stage into your final image just the module without the build tools/headers etc.

Any ideas why them should not be included in the BalenaOS by default?

Size. BalenaOS is built for embedded/small-spec machines, so wasting space on headers which aren’t needed at runtime wouldn’t make sense.

I mean seriously? No one runs wireguard on balena?

Not that I am aware of. I appreciate your frustration but let’s try and keep it constructive

drew · June 18, 2020, 4:44pm

Wow, thanks, that’s something… Sorry for being stupid, I am not a Docker pro, but It’s still not clear…

Let’s summarize what we’ve got here:

First Stage of my Dockerfile should be based on balenalib/%%BALENA_MACHINE_NAME%%-debian (so it pulls the headers and builds the needed kernel module). I guess HERE they explain where to get the code needed and how to build it.

Second Stage should be based on official linuxserver/wireguard docker image, but will also contain the needed by wireguard linux kernel module (I will copy it from shared filesystem between docker stages during second stage execution)? Therefore, the kernel module will be located in the resulted wireguard container instead of Host OS? Will it work at all?

Please let me know if I get it wrong or right.

Thank you

saintaardvark · June 18, 2020, 7:02pm

Hi there – thanks for your question. The procedure you’ve outlined is broadly correct, and you can read more about multi-stage docker builds at https://docs.docker.com/develop/develop-images/multistage-build/. As you say, the intent is to copy the kernel module to the second stage container (in your case, built on linuxserver/wireguard). This is the same procedure that’s used to support things like different Raspberry Pi Hats, so it should work here as well.

Hope that helps, and please let us know how you get on!

All the best,
Hugh

drew · June 20, 2020, 8:05pm

Ok, so I tried to figure this out, and this is what I ended up with so far (and honestly, I doubt that it’s gonna go further without other people’s help, lol)…

Dockerfile.template:

FROM balenalib/%%BALENA_MACHINE_NAME%%-debian

# Shouldn't this be ARG though? (Not sure)
ENV VERSION 'balenaOS 2.38.0+rev1'

WORKDIR /usr/src/app

RUN apt-get update && apt-get install -y curl wget build-essential \
    libelf-dev awscli bc flex libssl-dev python bison git pkg-config

ADD https://raw.githubusercontent.com/balena-os/kernel-module-build/master/build.sh .
ADD https://raw.githubusercontent.com/balena-os/kernel-module-build/master/workarounds.sh .

# get wireguard kernel module source code (as described here https://www.wireguard.com/compilation/):
RUN git clone https://git.zx2c4.com/wireguard-linux-compat

RUN BALENA_MACHINE_NAME=%%BALENA_MACHINE_NAME%% ./build.sh build \
      --device %%BALENA_MACHINE_NAME%% \
      --os-version "$VERSION" \
      --src wireguard-linux-compat/src


FROM linuxserver/wireguard
COPY --from=0 "/usr/src/app/wireguard-linux-compat/src/${AND_WHAT_DO_I_EXPECT_TO_COPY?}" /lib/modules/

Even if I knew the last piece of puzzle, which is this variable: "${AND_WHAT_DO_I_EXPECT_TO_COPY?}", I doubt it would work, as the wireguard instruction tells to install the module this way:

$ make -C wireguard-linux-compat/src -j$(nproc)
$ sudo make -C wireguard-linux-compat/src install

But I am trying the blackbox(to me lol) called build.sh

The problem here is that I honestly have never built any kernel modules, no clue what this kernel module is. Is it just a binary? Where the makefile will put it then? Does the makefile do anything else for it to work… I think this “simplistic wireguard VPN” setup (as it’s being advertised) should definitely not be such a hassle (hehe)…

Just leaving this here in case me, or more likely, someone smarter, becomes bored or inspired one day to finish this…

klutchell · July 11, 2020, 4:33pm

I found a working fork for raspberry pi 3/4 (and I don’t think I see the author’s name in this thread yet).

I’m working on something similar and will post it once I finish.

klutchell · July 11, 2020, 6:24pm

As promised, here’s my working project with links and acknowledgements for the scripts I borrowed.

drew · September 30, 2020, 12:52pm

That is so cool! Thank you very much for it!! Would it work on Pi Zero btw?

klutchell · September 30, 2020, 1:08pm

In theory, but you would have to specify the correct VERSION string for your OS to match your BALENA_MACHINE_NAME.

Also note that there is an official blog post now with similar instructions!

drew · October 1, 2020, 6:24am

Just done some testing… It works on pi4 with 2.56.0+rev1.dev (It even builds if specify wrong version).

It does not build on pi3 with 2.38.0+rev1.dev and on pi_zero with 2.48.0+rev1.dev (exact same output):

[wireguard]       Step 9/14 : FROM linuxserver/wireguard:v1.0.20200513-ls22
[wireguard]        ---> 47778cf8c7b9
[wireguard]       Step 10/14 : COPY --from=build /usr/src/kernel_modules_headers/ /usr/src/kernel_modules_headers/
[wireguard]        ---> 976e5fe63c03
[wireguard]       Step 11/14 : RUN make -C /usr/src/kernel_modules_headers M=/app/wireguard-linux-compat/src -j"$(nproc)"
[wireguard]        ---> Running in cacad2a493a2
[wireguard]       make: Entering directory '/usr/src/kernel_modules_headers'
[wireguard]         AR      /app/wireguard-linux-compat/src/built-in.o
[wireguard]         CC [M]  /app/wireguard-linux-compat/src/main.o
[wireguard]         CC [M]  /app/wireguard-linux-compat/src/noise.o
[wireguard]       gcc: error: unrecognized argument in option ‘-mabi=aapcs-linux’
[wireguard]       gcc: note: valid arguments to ‘-mabi=’ are: ilp32 lp64
[wireguard]
[wireguard]         CC [M]  /app/wireguard-linux-compat/src/device.o
[wireguard]       gcc: error: unrecognized argument in option ‘-mabi=aapcs-linux’
[wireguard]       gcc: note: valid arguments to ‘-mabi=’ are: ilp32 lp64
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mapcs’; did you mean ‘--specs’?
[wireguard]
[wireguard]         CC [M]  /app/wireguard-linux-compat/src/peer.o
[wireguard]       gcc: error: unrecognized argument in option ‘-mabi=aapcs-linux’
[wireguard]       gcc: note: valid arguments to ‘-mabi=’ are: ilp32 lp64
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mapcs’; did you mean ‘--specs’?
[wireguard]
[wireguard]         CC [M]  /app/wireguard-linux-compat/src/timers.o
[wireguard]       gcc: error: unrecognized argument in option ‘-mabi=aapcs-linux’
[wireguard]       gcc: note: valid arguments to ‘-mabi=’ are: ilp32 lp64
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mno-sched-prolog’; did you mean ‘-Wno-sign-promo’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mapcs’; did you mean ‘--specs’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mfpu=vfp’; did you mean ‘-mcpu=’?
[wireguard]
[wireguard]       gcc: error: unrecognized argument in option ‘-mabi=aapcs-linux’
[wireguard]       gcc: note: valid arguments to ‘-mabi=’ are: ilp32 lp64
[wireguard]
[wireguard]         CC [M]  /app/wireguard-linux-compat/src/queueing.o
[wireguard]       gcc: error: unrecognized command line option ‘-mapcs’; did you mean ‘--specs’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mno-sched-prolog’; did you mean ‘-Wno-sign-promo’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-msoft-float’
[wireguard]
[wireguard]         CC [M]  /app/wireguard-linux-compat/src/send.o
[wireguard]       gcc: error: unrecognized command line option ‘-mfpu=vfp’; did you mean ‘-mcpu=’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mapcs’; did you mean ‘--specs’?
[wireguard]
[wireguard]       scripts/Makefile.build:326: recipe for target '/app/wireguard-linux-compat/src/main.o' failed
[wireguard]       make[1]: *** [/app/wireguard-linux-compat/src/main.o] Error 1
[wireguard]       gcc: error: unrecognized command line option ‘-mno-sched-prolog’; did you mean ‘-Wno-sign-promo’?
[wireguard]       make[1]: *** Waiting for unfinished jobs....
[wireguard]
[wireguard]       gcc: error: unrecognized argument in option ‘-mabi=aapcs-linux’
[wireguard]       gcc: note: valid arguments to ‘-mabi=’ are: ilp32 lp64
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-msoft-float’
[wireguard]
[wireguard]         CC [M]  /app/wireguard-linux-compat/src/receive.o
[wireguard]       gcc: error: unrecognized command line option ‘-mfpu=vfp’; did you mean ‘-mcpu=’?
[wireguard]
[wireguard]       scripts/Makefile.build:326: recipe for target '/app/wireguard-linux-compat/src/noise.o' failed
[wireguard]       gcc: error: unrecognized argument in option ‘-mabi=aapcs-linux’
[wireguard]       gcc: note: valid arguments to ‘-mabi=’ are: ilp32 lp64
[wireguard]       gcc: error: unrecognized command line option ‘-mno-sched-prolog’; did you mean ‘-Wno-sign-promo’?
[wireguard]
[wireguard]       make[1]: *** [/app/wireguard-linux-compat/src/noise.o] Error 1
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-msoft-float’
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mfpu=vfp’; did you mean ‘-mcpu=’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mapcs’; did you mean ‘--specs’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mno-sched-prolog’; did you mean ‘-Wno-sign-promo’?
[wireguard]
[wireguard]       gcc: error: unrecognized argument in option ‘-mabi=aapcs-linux’
[wireguard]       gcc: note: valid arguments to ‘-mabi=’ are: ilp32 lp64
[wireguard]
[wireguard]       scripts/Makefile.build:326: recipe for target '/app/wireguard-linux-compat/src/device.o' failed
[wireguard]       make[1]: *** [/app/wireguard-linux-compat/src/device.o] Error 1
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mfpu=vfp’; did you mean ‘-mcpu=’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mapcs’; did you mean ‘--specs’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-msoft-float’
[wireguard]
[wireguard]       make[1]: *** [/app/wireguard-linux-compat/src/peer.o] Error 1
[wireguard]
[wireguard]       scripts/Makefile.build:326: recipe for target '/app/wireguard-linux-compat/src/peer.o' failed
[wireguard]       gcc: error: unrecognized command line option ‘-msoft-float’
[wireguard]
[wireguard]       scripts/Makefile.build:326: recipe for target '/app/wireguard-linux-compat/src/timers.o' failed
[wireguard]       make[1]: *** [/app/wireguard-linux-compat/src/timers.o] Error 1
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mapcs’; did you mean ‘--specs’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mno-sched-prolog’; did you mean ‘-Wno-sign-promo’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mfpu=vfp’; did you mean ‘-mcpu=’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mno-sched-prolog’; did you mean ‘-Wno-sign-promo’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mfpu=vfp’; did you mean ‘-mcpu=’?
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-msoft-float’
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-msoft-float’
[wireguard]       gcc: error: unrecognized command line option ‘-mno-sched-prolog’; did you mean ‘-Wno-sign-promo’?
[wireguard]
[wireguard]       scripts/Makefile.build:326: recipe for target '/app/wireguard-linux-compat/src/queueing.o' failed
[wireguard]       make[1]: *** [/app/wireguard-linux-compat/src/queueing.o] Error 1
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-mfpu=vfp’; did you mean ‘-mcpu=’?
[wireguard]
[wireguard]       scripts/Makefile.build:326: recipe for target '/app/wireguard-linux-compat/src/send.o' failed
[wireguard]       make[1]: *** [/app/wireguard-linux-compat/src/send.o] Error 1
[wireguard]
[wireguard]       gcc: error: unrecognized command line option ‘-msoft-float’
[wireguard]
[wireguard]       scripts/Makefile.build:326: recipe for target '/app/wireguard-linux-compat/src/receive.o' failed
[wireguard]       make[1]: *** [/app/wireguard-linux-compat/src/receive.o] Error 1
[wireguard]
[wireguard]       Makefile:1535: recipe for target '_module_/app/wireguard-linux-compat/src' failed
[wireguard]       make: *** [_module_/app/wireguard-linux-compat/src] Error 2
[wireguard]
[wireguard]       make: Leaving directory '/usr/src/kernel_modules_headers'
[wireguard]       Removing intermediate container cacad2a493a2
[wireguard]       The command '/bin/sh -c make -C /usr/src/kernel_modules_headers M=/app/wireguard-linux-compat/src -j"$(nproc)"' returned a non-zero code: 2
[Info]            Uploading images
[Success]         Successfully uploaded images
[Error]           Some services failed to build:
[Error]             Service: wireguard
[Error]               Error: The command '/bin/sh -c make -C /usr/src/kernel_modules_headers M=/app/wireguard-linux-compat/src -j"$(nproc)"' returned a non-zero code: 2
[Info]            Built on arm03
[Error]           Not deploying release.
Remote build failed

Additional information may be available by setting a DEBUG=1 environment
variable: "set DEBUG=1" on a Windows command prompt, "$env:DEBUG = 1" on
powershell, or "export DEBUG=1" on Linux or macOS.

If you need help, don't hesitate in contacting our support forums at
https://forums.balena.io

For CLI bug reports or feature requests, have a look at the GitHub issues or
create a new one at: https://github.com/balena-io/balena-cli/issues/

weird(

klutchell · October 1, 2020, 11:45am

For pi3 you could try hardcoding the architecture in the FROM linuxserver/wireguard:... field to FROM linuxserver/wireguard:arm32v7-latest because it looks like the arm64v8 arch was pulled by default.

gcc: error: unrecognized argument in option ‘-mabi=aapcs-linux’
[wireguard]       gcc: note: valid arguments to ‘-mabi=’ are: ilp32 lp64

I don’t think the same solution will work for pi-zero because linuxserver doesn’t have an armhf architecture but you can try with arm32v7 and see if it builds anyway. Also, I haven’t tried any dev kernel header packages myself so there may be other differences I’m not aware of.

You may also run into this issue that I don’t have a solution for quite yet.

5less · September 14, 2021, 11:24am

@klutchell Can I use your repository with Balena on Raspberry Pi 4?

klutchell · September 14, 2021, 11:42am

@5less Not at the moment, as last time I tried I had some issues getting the module to compile with the 64-bit kernel. I have some changes planned though that may help with this, so I will definitely be trying again when I have the time.

For now I’ve opened an issue for tracking and you can subscribe to notifications and updates: https://github.com/klutchell/balena-wireguard/issues/14

5less · September 14, 2021, 12:45pm

Thanks @klutchell, I will test and get back to you in the issue if I have anything of value to share.

Topic		Replies	Views
Wireguard client in container? balenaOS	9	1524	September 16, 2020
How to run Wireguard VPN in balenaOS Discussions	1	443	November 19, 2021
Using Wireguard with new OS releases Product support	22	1421	December 20, 2021
No Handshake while using klutchell/balena-wireguard Product support raspberrypi4	6	383	July 8, 2022
Trouble Getting balena run Command to Work Product support raspberrypi3	5	328	January 28, 2020

Is it impossible for now to run Wireguard VPN server container on BalenaOS?

Related topics