I see some references to successful deployment of 802.1x (eduroam etc) on BalenaOS WiFi interfaces but i was wondering if anyone has had some success testing this over the Ethernet interface (interested in RPi and NUC experiences). Of particular interest is the handling of client certificates for EAP-TLS based networks.
Hi @cam415, I never tested EAP over Ethernet - only over WiFi, but it should all be similar. From what I remember some enterprise modes required passing certificate(s) indeed. Those as far as I remember could be either embedded in the NetworkManager connection profile as string, or they can exist side-by-side with the connection profile on the filesystem.
It is probably the easiest approach to embed those in the connection profile file, but if that does not work for some reason (as I have some vague memory of me facing problems with that) you can have the certificate live in the same folder as the connection profile.
On balenaOS we have this mechanism where you place NetworkManager connection profiles in the system-connections folder on the boot partition and later on those are copied on boot to the proper location at /etc/NetworkManager/system-connections. This mechanism exists as the boot partition is FAT and you can open it from any OS while writing the image. Anyway, pointing you to that folder there as you may put the certificate there together with the connection profile. For more information on the system-connections folder you may check this section which includes that info: https://www.balena.io/docs/reference/OS/network/2.x/#changing-the-network-at-runtime
For experimenting you may always just modify files through a host OS terminal in the /etc/NetworkManager/system-connections directly and restart NetworkManager when modifications are done with systemctl restart NetworkManager. The boot partition approach is more important for producing production images.
As for the specifics of what settings will be needed for your authentication type this is an important reference page always: https://developer.gnome.org/NetworkManager/stable/nm-settings.html and you may internet search for EAP-TLS and NetworkManager together.
You may need to enable debug logging in case something is not working, and you may do that with:
dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager /org/freedesktop/NetworkManager org.freedesktop.NetworkManager.SetLogging string:"debug" string:""
Please let me know if you have any further questions.
Thanks,
Zahari
@majorz thanks for your detailed response. Took me a while to get a verified EAP-TLS lab environment up and running. Back into testing now with the NetworkManager profiles and I have been finding that creating / modifying files in the /etc/NetworkManager/system-connections location and restarting NetworkManager doesn’t appear to be applying the changes without a reboot of the device. Not a big deal but I suspect there will be a lot of tweaks to the profile required as I step through this process and it would be great to do some live testing. Let me know if you have any thoughts on why it is not picking up the profile files on restart. Thanks
My suspicion would be that the wpa_supplicant service that runs in the background (which is started by NetworkManager) is not clearing its state properly for this type authentication. Glad to hear that you made a good progress with this!
Thanks,
Zahari
@majorz With the need to reboot for the changes to take affect, have you got any suggestions on how to take advantage of the debug logging you suggested above. Essentially a black box until I get connectivity back to the box at the moment. The documentation on NetworkManager for including client certificates seems to be a little light on working examples (inband string (PEM or DER) vs host path) etc.
Try to kill wpa_supplicant before starting NetworkManager again. E.g. systemctl stop NetworkManager, ps|grep wpa_supplicant to get the PID of the process, kill PID and then systemctl start NetworkManager. This should eliminate the need to reboot the device. You may automate this by parsing the ps|grep output.
I looked at the NM sources and found this example which relates to your last question: https://github.com/NetworkManager/NetworkManager/blob/master/src/settings/plugins/keyfile/tests/keyfiles/Test_Wired_TLS_Blob
And here is the function that parses the certificates which could be worth checking: https://github.com/NetworkManager/NetworkManager/blob/52dbab7d0703d9f3982fcddd855b92886eaef866/shared/nm-keyfile/nm-keyfile.c#L1401
Thanks @majorz for the link to the TLS example - shame they are using the byte array option as this will be tougher to inject than just specifying a path to a PEM encoded file (which is the path I am pursuing). I think without access to the logging you suggested above this is going to be tough to troubleshoot. Its not clear to me where NetworkManager in BalenaOS is outputting those debog logs you were enabling with dbus. Any chance you can point me to the location.
Also if I want to inject the certificate files into the hostOS to read from the NM profile, is resin-boot the place to put them or will they need the .ignore extension if they are going to be copied over to the system-connections location?
From the sources used by the relevant function I see that it supports a few schemes (identified by prefixes):
#define NM_KEYFILE_CERT_SCHEME_PREFIX_PATH "file://"
#define NM_KEYFILE_CERT_SCHEME_PREFIX_PKCS11 "pkcs11:"
#define NM_KEYFILE_CERT_SCHEME_PREFIX_BLOB "data:;base64,"
Since PEM is base64 encoded my guess is that you can use it there after some modifications - join new lines and put the prefix in front.
For debugging you may run dbus-send --system --print-reply --dest=org.freedesktop.NetworkManager /org/freedesktop/NetworkManager org.freedesktop.NetworkManager.SetLogging string:"debug" string:"" from the host OS. Afterwards you can follow the logs with journalctl -u NetworkManager -f in a separate terminal.
You won’t need to put an .ignore extension to the file, as the file will be ignored by NetworkManager if it cannot be parsed as connection profile.
Thanks,
Zahari
Hi @majorz. I managed to make some progress over the weekend. Unfortunately I found that the config load of new setting from NetworkManager to wpa_supplicant appeared to only work on reboot so my ability to leverage the more detailed debugging was a little more difficult.
I was able to reference the certificate material just using the basic file path but I did end up just appending the PEM encoded file with .ignore just to be sure it wasn’t creating a conflict. I was a time consuming process making changes by mounting the flash card and trying to watch basic logs on the HDMI console.
Nonetheless I believe I have a basis of a working solution. What is your recommendation for injecting the certificate key material. Is my only option at time of flashing the SD card or is there an option to run a certificate enrollment workflow from a workload container and push this down to the host OS file system?
Thanks for your great support!
With the file-based approach this cannot be done from a container. With a container it can only be possible for the embedded blob approach through the NM API.
Just for clarity, you are referring to driving the NetworkManager API through the container DBUS connection similar to what is shown in this example:
But to your point using the blob format inline so it can be passed from container to host OS using this method.
Yes, and there are higher level wrapper libraries like python-networkmanager which deal with the dbus internally.
If anyone else lands here looking for a Ethernet based dot1x profile, here is an example connection profile for NetworkManager:
[connection]
id=my-ethernet
type=ethernet
interface-name=eth0
permissions=
secondaries=
[ethernet]
mac-address-blacklist=
[802-1x]
altsubject-matches=
eap=tls;
identity=rpi
ca-cert=/etc/NetworkManager/system-connections/ca-cert.pem
client-cert=/etc/NetworkManager/system-connections/rpi.pem
private-key=/etc/NetworkManager/system-connections/rpi.p12
private-key-password=p12pwd
[ipv4]
dns-search=
method=auto
[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto
Note: the rpi.pem included the device certificate and the entire trust chain to the self signed Root CA.