Networking Setup: Raspberry Pi as a Router

Hey there,

first off, merry belated christmas to you all, I hope you had a nice christmas week.

I’m currently trying out Balena for an IoT-ish project and at this point I’ve happened across some networking issues that I seem incapable of resolving on my own. I’ve never worked with NetworkManager before, so I’m struggling a bit to wrap my head around getting it to do what I want it to, so I figured I’d ask for help here.

My setup is as follows:

  • I have multiple Balena-imaged devices with network connectivity (Ethernet)
  • I have one “main” device (also Balena-imaged) which has multiple network interfaces

The main device is a Raspberry Pi 4B. It has two ethernet ports (one via a USB adapter for the time being) and WiFi. The other devices have one ethernet port each.

I would like to set the Pi up such that it can intake an existing internet connection through some means (either one of the Ethernet ports or possibly a GSM modem later down the line) and subsequently act as a router for all “downstream” devices. A switch is hooked up to one of the ethernet ports, and several more devices are connected to that switch.

The end goal I have in mind would be for the Pi to work as a “router” in the sense that all downstream devices and the WiFi would be in one network, with internet connectivity supplied to the Pi through a secondary interface. Let us assume this is another ethernet port to make this more concrete.

I’ve attached an image showing what I have in mind below.

I already have some progress towards this goal, but it’s not quite there yet.

I first created a WiFi hotspot according to Balena’s networking documentation:

[connection]
id=balena-hotspot
type=wifi
autoconnect=true
interface-name=wlan0
permissions=
secondaries=

[wifi]
band=bg
mac-address-blacklist=
mac-address-randomization=0
mode=ap
seen-bssids=
ssid=WiFi

[wifi-security]
group=
key-mgmt=wpa-psk
pairwise=
proto=rsn
psk=123456789

[ipv4]
dns-search=
method=shared

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=ignore

Then I found a setup that somewhat worked for the ethernet ports:

[connection]
id=eth-bridge
type=ethernet
interface-name=eth0
permissions=

[ethernet]
mac-address-blacklist=

[ipv4]
dns-search=
method=shared

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=ignore

[proxy]

Truth be told, I don’t exactly understand what this does, but it ended up with me being able to see some of the downstream devices when connected to the hotspot. However, the devices themselves did not connect to the outside world. I did have internet access through the hotspot though. It was quite strange (to me at least). It was like I was looking at the network through the eyes of my main device, not as a client connected to it.

I also noticed that in Balena Dashboard, the Raspberry Pi showed up with multiple IP addresses, presumably because of the hotspot. One IP was for the incoming internet connection, one for the Hotspot. There was also a third one in yet another subnet which I couldn’t pinpoint.

The closest I got for a similar setup was this: https://raspberrypi.stackexchange.com/a/99598
Alas, it uses systemd networking, which is different from NetworkManager, I think, so does not translate properly to Balena’s environment.

As you can probably tell, this is a mess, so I’m hoping that somebody who knows more about working with NetworkManager than me can give me some pointers on:

  • How to set this up
  • Whether this setup is feasible
  • Whether this setup is a good idea at all or whether something else may work better
  • Whether my current approach is anywhere close to going into the right direction

Have a good one, hopefully someone can help me out with this :slight_smile:

I’ve never tried that myself, but it sounds theoretically possible. What you need first is to combine the wireless and wired interfaces in one additional interface and set method=shared on top of it. You can try with bond or bridge interfaces and see which one will work for you. The wireless and wired interfaces will be slave interfaces, where the new additional one will be a master interface.

Okay, I’ve made some progress on this. That comment about bonding eventually led me to a somewhat working configuration thanks to some documentation I found.

My current setup looks as follows.

Bridge:

[connection]
id=bridge
type=bridge
interface-name=mybridge

[ipv4]
addresses=192.167.0.1/24
method=shared

[ipv6]
method=ignore

Ethernet:

[connection]
id=bridge-eth
type=ethernet
interface-name=eth0
master=mybridge
slave-type=bridge

Access Point:

[connection]
id=bridge-wlan
type=wifi
interface-name=wlan0
autoconnect=true
master=mybridge
slave-type=bridge

[wifi]
mode=ap
ssid=MySSID

With this setup, I can connect devices to both wlan0 as well as a switch behind eth0 and I can see them from either end.

Two conundrums that remain:

  1. I couldn’t figure out how to get WiFi authentication to work properly yet
  2. I do not have internet connectivity. While I can roam around inside the bridge network, I can not call out to anywhere

Any pointers regarding internet sharing with such a bridged setup?

According to the RedHat documentation on NetworkManager, this setup should automatically hook up the bridge to the default network interface. Mine is (I assume) correctly labeling the USB ethernet with the internet connection as ipv4 default, so that should be fine, I hope. Still, no actual connection. I’m wondering what is interfering with the NAT here.

Hi, regarding WiFi security, you probably need a wifi-security section in your configuration file too, something like:

[wifi-security]
key-mgmt=wpa-psk
psk=<password>

Depending on the type of authentication you want to use the above might need tweaking.

Regarding internet connectivity, do you have a default gateway appear in your routing table? Check the output of the route command.

If not, try setting a gateway address in your ipv4 configuration. Check the NetworkManager documentation for details https://developer.gnome.org/NetworkManager/stable/settings-ipv4.html.

Thank you! The gateway was what did it. Adding it gave me full internet connectivity for downstream devices. nmcli rendered it into this configuration format:

[ipv4]
address1=192.167.0.1/24,192.167.0.1
method=shared

Regarding the security, unfortunately I’m having some trouble with that. If I set the security up like this:

[wifi-security]
group=
key-mgmt=wpa-psk
pairwise=
proto=rsn
psk=123456789

I lose the ability to connect to the AP. All connection attempts just boot me back to the “Enter Password” prompt, as if I had entered a wrong password. This happens on macOS as well as on my Android phone, which I tested with as a sanity check. It behaves as if the password is not 123456789. This is quite puzzling, because I could’ve sworn this exact config worked standalone, as it’s effectively the same as the Hotspot example from the Balena Networking docs.
journalctl doesn’t really output any useful information when these connection attempts happen, so I’m kind of at a loss here.

We’re getting closer though, thank you for the help!

Hi, glad it helped. About the WiFi security, it’s better to configure the minimum and let NM flexibility to negotiate the rest, otherwise you might be forcing something that is non-negotiable. Try just with:

[wifi-security]
key-mgmt=wpa-psk
psk=<password>

Also, you can try to increase NetworkManager’s verbosity as explained in https://www.balena.io/docs/learn/more/masterclasses/host-os-masterclass/#52-making-networkmanager-logs-more-verbose-in-four-ways

Alas, even the elevated debug level outputs nothing that seems directly related to the connection issue (in combination with the config you proposed). The only thing I can see happening when attempting to connect to the AP is the line wlan0: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0 being output by wpa_supplicant, which should be correct. Strangely though I also get this exact message if I submit an incorrect password, whereas this entry here has led me to believe that it should be different.

There’s quiet a few somewhat similar topics to be found via a cursory google search, a lot of which pointed the finger at wpa_supplicant, either by downgrading/upgrading it or it having a strange config, but I’m not sure whether that applies here.

I’m also seeing this somewhat puzzling output:

root@<device-id>:~# nmcli device wifi list
IN-USE  SSID       MODE   CHAN  RATE      SIGNAL  BARS  SECURITY  
*       MySSID     Infra  1     0 Mbit/s  0             WPA1 WPA2 

Maybe the rate of 0 has something to do with this?

Is it possible that my config is not taking entirely? That shouldn’t be Infra, as far as I can tell.

Edit

Okay, did a quick sanity check with the non-bridge AP, and that works flawlessly:

wpa_supplicant[1587]: wlan0: CTRL-EVENT-SUBNET-STATUS-UPDATE status=0
wpa_supplicant[1587]: wlan0: AP-STA-CONNECTED MAC-ADDRESS
dnsmasq-dhcp[1882]: DHCPDISCOVER(wlan0) MAC-ADDRESS
dnsmasq-dhcp[1882]: DHCPOFFER(wlan0) 10.42.0.162 MAC-ADDRESS
dnsmasq-dhcp[1882]: DHCPDISCOVER(wlan0) MAC-ADDRESS
dnsmasq-dhcp[1882]: DHCPOFFER(wlan0) 10.42.0.162 MAC-ADDRESS
dnsmasq-dhcp[1882]: DHCPREQUEST(wlan0) 10.42.0.162 MAC-ADDRESS
dnsmasq-dhcp[1882]: DHCPACK(wlan0) 10.42.0.162 MAC-ADDRESS
dnsmasq-dhcp[1882]: DHCPREQUEST(wlan0) 10.42.0.162 MAC-ADDRESS
dnsmasq-dhcp[1882]: DHCPACK(wlan0) 10.42.0.162 MAC-ADDRESS
dnsmasq-dhcp[1882]: DHCPREQUEST(wlan0) 10.42.0.162 MAC-ADDRESS
dnsmasq-dhcp[1882]: DHCPACK(wlan0) 10.42.0.162 MAC-ADDRESS

I can connect and everything works just fine. The config used was taken from the Balena docs here and looks like this:

[connection]
id=balena-hotspot
type=wifi
autoconnect=true
interface-name=wlan0

[wifi]
mode=ap
ssid=MySSID

[wifi-security]
key-mgmt=wpa-psk
psk=123456789

[ipv4]
method=shared

[ipv6]
method=ignore

So how can it be that when I add

master=mybridge
slave-type=bridge

to this config, that the connection suddenly breaks? Is this a documented incompatibility that is not supposed to work, or is this an issue that might span a bit more than just my particular usecase?

Edit

Doing a bit more digging today and I came across this PR to NetworkManager, which seems to indicate that in a configuration where a WiFi AP is slaved to a bridge interface, NM may not properly pass through the interface name of the bridge, which could potentially cause this issue. This is also corroborated here.

I see that the NM on the Balena image is version 1.20.2:

root@<device-id>:~# nmcli -v    
nmcli tool, version 1.20.2

This version was released in September 2019, so I assume updating to a newer version might work. Thoughts on this?

Not sure if double posting/bumping is allowed (please let me know if it isn’t). I added some information to my previous post and I’m not sure whether that generates a notification or not, so this post is meant to trigger that if it didn’t.

I don’t have any further ideas to share regarding your debug quest, but have you taken a look at the methods used in our AP/Repeater project here? I’m wondering if it may generate some ideas: https://github.com/balenalabs-incubator/wifi-repeater

Thanks for the input, I appreciate it!
Unfortunately this project only makes somewhat rudimentary use of NetworkManager in such capacity as that multiple WiFi interfaces are activated or shared to an Ethernet connection. That part also works here.

The main problem as I currently see it lies within the specific behavior of NetworkManager in the case where a WiFi AP is slaved to a network bridge. In that particular case, wpa_supplicant can fail to initialize properly, causing non-connectivity for the WiFi AP.

As far as I was able to determine up until now, this may actually be a bug/strange behavior in NetworkManager itself, so I might be bitten here unless there’s some way I can get a more up-to-date version going. Is there some way I can upgrade it on the image myself? Can I download the sources and compile? Run a docker container with a newer version of NM?

If none of these are options, I think I will remove Balena from my Raspberry Pi and test on regular Ubuntu (with NM enabled). Hopefully I’ll be able to get a newer version there so that I can compare the behavior.

Since I can’t edit my post anymore I’ll post an update like this.
I tried compiling the latest master version of NM from source and using that to try and replicate my issue (in the hopes that the PR/issue I linked actually fixed this problem), but unfortunately I did not manage it. NM has an absolute boatload of dependencies and compilation takes ages on my Raspberry Pi, so after a few failed compilation attempts I gave up due to time constraints. The one time I did actually manage to get it to build it failed because some libraries weren’t properly linking at runtime. I did not have the strength of mind to debug go further down this rabbit hole at that point.

As such I’m posting my current progress in the faint hope that maybe someday, this will be resolved properly. As far as I can tell right now, this is in fact an issue with the old version of NM shipping on Balena images, and there’s very little a user can do about it save for building their own custom image (which I think is possible but likely troublesome, given my experience with compiling NM just now).

Anyway, the NM configuration files to get this router-AP business going are as follows:

Create a bridge:

[connection]
id=bridge
type=bridge
interface-name=mybridge

[ipv4]
address1=192.167.0.1/24,192.167.0.1
method=shared

[ipv6]
method=ignore

This needs to set the IP subnet and range as well as the gateway in the form shown above for internet connectivity to work properly.

Next, we can slave a free ethernet interface to the bridge:

[connection]
id=bridge-eth
type=ethernet
interface-name=eth0
master=mybridge
slave-type=bridge

Here I use the default one integrated into the Raspberry Pi as I have a USB ethernet dongle connected providing the actual internet connection.

Lastly, the AP can be configured thusly:

[connection]
id=bridge-wlan-no-sec
type=wifi
interface-name=wlan0
autoconnect=true
master=mybridge
slave-type=bridge

[wifi]
mode=ap
ssid=MySSID

In a perfect world, one would be able to add security to the AP like this:

[wifi-security]
key-mgmt=wpa-psk
psk=123456789

However, with the current situation, it is absolutely essential that the AP is unprotected as otherwise it will not allow any connections to it.

With this setup, if a secondary ethernet connection is established with an existing internet connection, everything downstream from the Raspberry Pi will see the same it would if it were an actual router serving this network. Downstream devices have their own subnet plus a route to the internet and can see each other between WiFi and Ethernet.

If anybody else comes across this thread and has any insight to share whatsoever, please by all means go ahead, I’m still looking for a fully working solution enabling security, so feel free to share anything that might help. I’ll update this thread with a solution should I ever find one.

Thanks for the detailed upgrade. We are going to release an updated version of NetworkManager in upcoming balenaOS release. It is scheduled already and usually it takes a few weeks to reach production. Before that it reaches staging though: https://dashboard.balena-staging.com/. You may check in a couple of weeks to see whether it is available.

Sweet, thanks a ton for being this responsive on the issue!
I’ll certainly check this out in a few weeks time and let you know of the outcome.
Thanks again!

Hello @byteminer, NetwokManager will be updated to 1.28.0 in balenaOS 2.68.0: networkmanager: Update to 1.28.0 by majorz · Pull Request #2091 · balena-os/meta-balena · GitHub
A 2.68.0 raspberry pi image has not been released yet.

Thanks for the update!
I’ll investigate once an image is available and post my findings here.