Routing problem when setting up shared access point

Would you mind sharing your docker setup so I can try out a few things this weekend?

1 Like

I use this layer with some smaller changes: https://github.com/srednak/balena-st-stm32mp

You can find my networkmanager connection configuration above in this thread.

The problem is that I don’t think you can replicate the collision I got as it was gone after I added my script-file and service that will start the script without enabling it. Balena firewall still remove those INPUT rules, but internet sharing is working (If you use other DNS server than default 192.168.1.1.

It’s only if networkmanager can’t create those FORWARD rules, internet sharing will not work.

Hi,

I played around a bit today.
It was quite easy to replicate NetworkManager not being able to add the routes.

Just need a shared interface and a terminal in the HostOS.

  1. grab the lockfile (ex. flock /run/xtables.lock -c "sleep 120s")
  2. bring up the shared connection (nmcli con up balena-hotspot)

I’ve also tried this with multiple shared connections (after adding dummy interfaces), and it seems NetworkManager properly handles trying to bring them up simultaneously.

I’ve had a look at NetworkManager dispatcher scripts, but you can’t halt the main process from those because they are spawned in a separate process. So while your script there may wait for the lockfile to become accessible (and prevent the up command from returning), the iptables commands will still run (and fail) before your script finishes.

For the balena-enginepart, I tried balena network create test, which adds some rules to the nat table.
It was indeed quite difficult to get this to occur at the same time as the NetworkManager bit.
However, running the command while holding the lockfile, showed the balena command waits properly for the lock to be released before adding the rules.

It seems like you were able to consistently recreate the balena/NetworkManager timing.
So something I’m curious about is if you can create a wrapper script for iptables in the Host OS to see if the wait flag actually fixes this.

For example, you could replace the /usr/sbin/iptables symbolic link with the following script:

#!/bin/sh
/usr/sbin/xtables-legacy-multi iptables -w 5  "$@"
1 Like

Thank you very much. I tried that trick with hijacking iptables symbol link, and it worked every time.
I have restart the devices multiple times. Now I only have to implement the hijack in my recipe.

When iptables look like this it usually does not work as it competes with balena-engine to add rules.

-A INPUT -j BALENA-FIREWALL
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o balena0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o balena0 -j DOCKER
-A FORWARD -i balena0 ! -o balena0 -j ACCEPT
-A FORWARD -i balena0 -o balena0 -j ACCEPT
-A FORWARD -d 192.168.1.0/24 -o wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.1.0/24 -i wlan0 -j ACCEPT
-A FORWARD -i wlan0 -o wlan0 -j ACCEPT
-A FORWARD -o supervisor0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o supervisor0 -j DOCKER
-A FORWARD -i supervisor0 ! -o supervisor0 -j ACCEPT
-A FORWARD -i supervisor0 -o supervisor0 -j ACCEPT
-A FORWARD -o br-a8483e87e992 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-a8483e87e992 -j DOCKER
-A FORWARD -o wlan0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i wlan0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i br-a8483e87e992 ! -o br-a8483e87e992 -j ACCEPT
-A FORWARD -i br-a8483e87e992 -o br-a8483e87e992 -j ACCEPT
-A BALENA-FIREWALL -m state --state RELATED,ESTABLISHED -j ACCEPT
-A BALENA-FIREWALL -m addrtype --src-type LOCAL -j ACCEPT
-A BALENA-FIREWALL -i resin-vpn -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i tun0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i docker0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i lo -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i supervisor0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -p tcp -m tcp --dport 48484 -j REJECT --reject-with icmp-port-unreachable
-A BALENA-FIREWALL -p tcp -m tcp --dport 22222 -j ACCEPT
-A BALENA-FIREWALL -p tcp -m tcp --dport 2375 -j ACCEPT
-A BALENA-FIREWALL -m addrtype --dst-type MULTICAST -j ACCEPT
-A BALENA-FIREWALL -p icmp -j ACCEPT
-A BALENA-FIREWALL -i balena0 -p udp -m udp --dport 53 -j ACCEPT
-A BALENA-FIREWALL -j RETURN
-A BALENA-FIREWALL -j REJECT --reject-with icmp-port-unreachable
-A DOCKER-ISOLATION-STAGE-1 -i balena0 ! -o balena0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i supervisor0 ! -o supervisor0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-a8483e87e992 ! -o br-a8483e87e992 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o balena0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o supervisor0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-a8483e87e992 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

Where is best place to put this modification in my Yocto layer?

I created a iptables_%.bbappend file and added following code:

# Include the iptables retry script
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"

SRC_URI_append = " \
    file://iptables-retry \
    "


# Install iptable retry script and hijack iptables symlink
do_install_append() {
    install -m 0755 ${WORKDIR}/iptables-retry ${D}/${sbindir}/iptables-retry
    ln -sf iptables-retry ${D}/${sbindir}/iptables
}

The script is like TjVv suggested:

#!/bin/sh
/usr/sbin/xtables-legacy-multi iptables -w 5  "$@"

The above solution is working but maybe there is a more neat way to do it.

What about changing

–with-iptables=${sbindir}/iptables

in the networkmanager recipie? Is it possible to add the -w 5 argument here?

I’m not quite sure about the place for to inject this workaround; will poke around tonight a bit.

I will say that before using this in the wild, the script will need a little input checking, for example to prevent a second -w or --wait argument.

1 Like

As we know networkmanager does not use --wait or -w we can put the installation on this workaround script in a bbappend for networkmanager instead and just change –with-iptables
In that way the workaround will only affect networkmanager.

Does supervisor container have iptables installed and run same command at same time?

How do I check that?

Life (including work) got in the way of looking at the yocto recipes.

Regarding iptables in supervisor, you can grab the sources from github and search through it.
You will see iptables is installed and used to allow access to VPN and supervisor.

I’ve had a look at injecting this and had some success it seems.
Based on your work, I made a recipes-connectivity/networkmanager/networkmanager_%.bbappend:

# Include the iptables retry script
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"

SRC_URI_append = " \
    file://fake-iptables.sh \
    "

# Install iptable retry script and hijack iptables symlink
do_install_append() {
    install -m 0755 ${WORKDIR}/fake-iptables.sh ${D}/${sbindir}/fake-iptables
}

EXTRA_OECONF += " --with-iptables=${sbindir}/fake-iptables"

The script it refers to is recipes-connectivity/networkmanager/files/fake-iptables.sh:

#!/bin/sh
echo "$@" >> /tmp/iptables.log
/usr/sbin/iptables "$@"

Building and flashing the image, shows me the following:

  1. iptables works normally as expected
  2. When I add a shared connection in NetworkManager, the relevant rules pop up in the log file I use in the fake-iptables script.
root@86a5d95:~# cat /tmp/iptables.log 
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 67 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 67 --jump ACCEPT
--table filter --insert FORWARD --in-interface wlan0 --jump REJECT
--table filter --insert FORWARD --out-interface wlan0 --jump REJECT
--table filter --insert FORWARD --in-interface wlan0 --out-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --source 192.168.1.0/255.255.255.0 --in-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --destination 192.168.1.0/255.255.255.0 --out-interface wlan0 --match state --state ESTABLISHED,RELATED --jump ACCEPT
--table nat --insert POSTROUTING --source 192.168.1.0/255.255.255.0 ! --destination 192.168.1.0/255.255.255.0 --jump MASQUERADE

Here you can see the normal iptables vs fake-iptables operation:

root@86a5d95:~# iptables --version
iptables v1.8.4 (legacy)
root@86a5d95:~# cat /tmp/iptables.log 
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 67 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 67 --jump ACCEPT
--table filter --insert FORWARD --in-interface wlan0 --jump REJECT
--table filter --insert FORWARD --out-interface wlan0 --jump REJECT
--table filter --insert FORWARD --in-interface wlan0 --out-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --source 192.168.1.0/255.255.255.0 --in-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --destination 192.168.1.0/255.255.255.0 --out-interface wlan0 --match state --state ESTABLISHED,RELATED --jump ACCEPT
--table nat --insert POSTROUTING --source 192.168.1.0/255.255.255.0 ! --destination 192.168.1.0/255.255.255.0 --jump MASQUERADE
root@86a5d95:~# fake-iptables --version
iptables v1.8.4 (legacy)
root@86a5d95:~# cat /tmp/iptables.log 
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 67 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 67 --jump ACCEPT
--table filter --insert FORWARD --in-interface wlan0 --jump REJECT
--table filter --insert FORWARD --out-interface wlan0 --jump REJECT
--table filter --insert FORWARD --in-interface wlan0 --out-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --source 192.168.1.0/255.255.255.0 --in-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --destination 192.168.1.0/255.255.255.0 --out-interface wlan0 --match state --state ESTABLISHED,RELATED --jump ACCEPT
--table nat --insert POSTROUTING --source 192.168.1.0/255.255.255.0 ! --destination 192.168.1.0/255.255.255.0 --jump MASQUERADE
--version

To me it looks like this way you can change it for specific packages, rather than globally.
The main question you have to ask now is whether you want the change globally or not.

1 Like

I’ve also tried this with multiple shared connections (after adding dummy interfaces), and it seems NetworkManager properly handles trying to bring them up simultaneously.

Is there any scripts in balena that adds iptables rules that can create the conflict.

.

I see no different in your outputs, but maybe that is your point.

I think the solutions is good enough. I will just change the script to use iptables instead of xtables-legacy-multi as follow:

#!/bin/sh
/usr/sbin/iptables -w 5  "$@"

My networkmanager_%.bbappend look like this:

# Include the iptables retry script
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"

SRC_URI_append = " \
    file://iptables-retry \
    "

# Install iptable retry script and hijack iptables symlink
do_install_append() {
    install -m 0755 ${WORKDIR}/iptables-retry ${D}/${sbindir}/iptables-retry
}

EXTRA_OECONF_remove = "--with-iptables=${sbindir}/iptables"
EXTRA_OECONF_append = " --with-iptables=${sbindir}/iptables-retry "

The iptables rules are now ok but I still have problems with my connected wifi clients. They report no internet sometimes but have no problems connecting to my server running in my container.
So how do I troubleshoot whats wrong?

Is there still problems in my routing like with the meter info or iptables rules or what else can it be?

My Huawei phone shows a Hotspot connected symbol when connecting to my Balena router while it show a Wi-fi network connected symbol when connecting to my home wifi router. I don’t understand what differ.

I got a suggestion that is all about metric information. If I get the Hotspot symbol, my phone handle the connection as a not unlimited wifi network.

I looked in the routing table in the begining of this thread and found some metric data. Can that be the problem and how do I fix it?

Wwan0 is the modem and wlan0 is my wifi acting as a Hotspot.

Okey I have found out what make that “Hotspot connected” symbol shown when connecting my phone to my device. It was the Network Manager connection.metered value on my wifi hotspot interface that was set as unknown. By setting the value to “no” will change the symbol to the regular “Wi-Fi network connected”.

I still have some intermittent problems with internet connection and I think the MTU value is the problem. My wwan0 connection (modem) got MTU set to 1430. By change MTU to 1400 on my wifi clients I can get it working.
As I understand the DHCP server can tell wifi clients to use a specific MTU value. How do I set this on my device? I can set 802-11-wireless.mtu to 1400 but the clients doesn’t care about it.

Got it working by alter the MSS (Maximum Segment Size) value of TCP SYN packets with:

iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o wwan0 -j TCPMSS --clamp-mss-to-pmtu

So, I guess I have to install iptables in the container and add this rule to it.