Would you mind sharing your docker setup so I can try out a few things this weekend?
I use this layer with some smaller changes: https://github.com/srednak/balena-st-stm32mp
You can find my networkmanager connection configuration above in this thread.
The problem is that I don’t think you can replicate the collision I got as it was gone after I added my script-file and service that will start the script without enabling it. Balena firewall still remove those INPUT rules, but internet sharing is working (If you use other DNS server than default 192.168.1.1.
It’s only if networkmanager can’t create those FORWARD rules, internet sharing will not work.
Hi,
I played around a bit today.
It was quite easy to replicate NetworkManager
not being able to add the routes.
Just need a shared interface and a terminal in the HostOS.
- grab the lockfile (ex.
flock /run/xtables.lock -c "sleep 120s"
) - bring up the shared connection (
nmcli con up balena-hotspot
)
I’ve also tried this with multiple shared connections (after adding dummy interfaces), and it seems NetworkManager properly handles trying to bring them up simultaneously.
I’ve had a look at NetworkManager dispatcher scripts, but you can’t halt the main process from those because they are spawned in a separate process. So while your script there may wait for the lockfile to become accessible (and prevent the up command from returning), the iptables commands will still run (and fail) before your script finishes.
For the balena-engine
part, I tried balena network create test
, which adds some rules to the nat
table.
It was indeed quite difficult to get this to occur at the same time as the NetworkManager
bit.
However, running the command while holding the lockfile, showed the balena command waits properly for the lock to be released before adding the rules.
It seems like you were able to consistently recreate the balena/NetworkManager timing.
So something I’m curious about is if you can create a wrapper script for iptables in the Host OS to see if the wait flag actually fixes this.
For example, you could replace the /usr/sbin/iptables
symbolic link with the following script:
#!/bin/sh
/usr/sbin/xtables-legacy-multi iptables -w 5 "$@"
Thank you very much. I tried that trick with hijacking iptables symbol link, and it worked every time.
I have restart the devices multiple times. Now I only have to implement the hijack in my recipe.
When iptables look like this it usually does not work as it competes with balena-engine to add rules.
-A INPUT -j BALENA-FIREWALL
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o balena0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o balena0 -j DOCKER
-A FORWARD -i balena0 ! -o balena0 -j ACCEPT
-A FORWARD -i balena0 -o balena0 -j ACCEPT
-A FORWARD -d 192.168.1.0/24 -o wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.1.0/24 -i wlan0 -j ACCEPT
-A FORWARD -i wlan0 -o wlan0 -j ACCEPT
-A FORWARD -o supervisor0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o supervisor0 -j DOCKER
-A FORWARD -i supervisor0 ! -o supervisor0 -j ACCEPT
-A FORWARD -i supervisor0 -o supervisor0 -j ACCEPT
-A FORWARD -o br-a8483e87e992 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-a8483e87e992 -j DOCKER
-A FORWARD -o wlan0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i wlan0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i br-a8483e87e992 ! -o br-a8483e87e992 -j ACCEPT
-A FORWARD -i br-a8483e87e992 -o br-a8483e87e992 -j ACCEPT
-A BALENA-FIREWALL -m state --state RELATED,ESTABLISHED -j ACCEPT
-A BALENA-FIREWALL -m addrtype --src-type LOCAL -j ACCEPT
-A BALENA-FIREWALL -i resin-vpn -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i tun0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i docker0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i lo -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i supervisor0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -p tcp -m tcp --dport 48484 -j REJECT --reject-with icmp-port-unreachable
-A BALENA-FIREWALL -p tcp -m tcp --dport 22222 -j ACCEPT
-A BALENA-FIREWALL -p tcp -m tcp --dport 2375 -j ACCEPT
-A BALENA-FIREWALL -m addrtype --dst-type MULTICAST -j ACCEPT
-A BALENA-FIREWALL -p icmp -j ACCEPT
-A BALENA-FIREWALL -i balena0 -p udp -m udp --dport 53 -j ACCEPT
-A BALENA-FIREWALL -j RETURN
-A BALENA-FIREWALL -j REJECT --reject-with icmp-port-unreachable
-A DOCKER-ISOLATION-STAGE-1 -i balena0 ! -o balena0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i supervisor0 ! -o supervisor0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-a8483e87e992 ! -o br-a8483e87e992 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o balena0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o supervisor0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-a8483e87e992 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
Where is best place to put this modification in my Yocto layer?
I created a iptables_%.bbappend file and added following code:
# Include the iptables retry script
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI_append = " \
file://iptables-retry \
"
# Install iptable retry script and hijack iptables symlink
do_install_append() {
install -m 0755 ${WORKDIR}/iptables-retry ${D}/${sbindir}/iptables-retry
ln -sf iptables-retry ${D}/${sbindir}/iptables
}
The script is like TjVv suggested:
#!/bin/sh
/usr/sbin/xtables-legacy-multi iptables -w 5 "$@"
The above solution is working but maybe there is a more neat way to do it.
What about changing
–with-iptables=${sbindir}/iptables
in the networkmanager recipie? Is it possible to add the -w 5 argument here?
I’m not quite sure about the place for to inject this workaround; will poke around tonight a bit.
I will say that before using this in the wild, the script will need a little input checking, for example to prevent a second -w
or --wait
argument.
As we know networkmanager does not use --wait or -w we can put the installation on this workaround script in a bbappend for networkmanager instead and just change –with-iptables
In that way the workaround will only affect networkmanager.
Does supervisor container have iptables installed and run same command at same time?
How do I check that?
Life (including work) got in the way of looking at the yocto recipes.
Regarding iptables in supervisor, you can grab the sources from github and search through it.
You will see iptables is installed and used to allow access to VPN and supervisor.
I’ve had a look at injecting this and had some success it seems.
Based on your work, I made a recipes-connectivity/networkmanager/networkmanager_%.bbappend
:
# Include the iptables retry script
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI_append = " \
file://fake-iptables.sh \
"
# Install iptable retry script and hijack iptables symlink
do_install_append() {
install -m 0755 ${WORKDIR}/fake-iptables.sh ${D}/${sbindir}/fake-iptables
}
EXTRA_OECONF += " --with-iptables=${sbindir}/fake-iptables"
The script it refers to is recipes-connectivity/networkmanager/files/fake-iptables.sh
:
#!/bin/sh
echo "$@" >> /tmp/iptables.log
/usr/sbin/iptables "$@"
Building and flashing the image, shows me the following:
-
iptables
works normally as expected - When I add a
shared
connection inNetworkManager
, the relevant rules pop up in the log file I use in thefake-iptables
script.
root@86a5d95:~# cat /tmp/iptables.log
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 67 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 67 --jump ACCEPT
--table filter --insert FORWARD --in-interface wlan0 --jump REJECT
--table filter --insert FORWARD --out-interface wlan0 --jump REJECT
--table filter --insert FORWARD --in-interface wlan0 --out-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --source 192.168.1.0/255.255.255.0 --in-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --destination 192.168.1.0/255.255.255.0 --out-interface wlan0 --match state --state ESTABLISHED,RELATED --jump ACCEPT
--table nat --insert POSTROUTING --source 192.168.1.0/255.255.255.0 ! --destination 192.168.1.0/255.255.255.0 --jump MASQUERADE
Here you can see the normal iptables
vs fake-iptables
operation:
root@86a5d95:~# iptables --version
iptables v1.8.4 (legacy)
root@86a5d95:~# cat /tmp/iptables.log
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 67 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 67 --jump ACCEPT
--table filter --insert FORWARD --in-interface wlan0 --jump REJECT
--table filter --insert FORWARD --out-interface wlan0 --jump REJECT
--table filter --insert FORWARD --in-interface wlan0 --out-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --source 192.168.1.0/255.255.255.0 --in-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --destination 192.168.1.0/255.255.255.0 --out-interface wlan0 --match state --state ESTABLISHED,RELATED --jump ACCEPT
--table nat --insert POSTROUTING --source 192.168.1.0/255.255.255.0 ! --destination 192.168.1.0/255.255.255.0 --jump MASQUERADE
root@86a5d95:~# fake-iptables --version
iptables v1.8.4 (legacy)
root@86a5d95:~# cat /tmp/iptables.log
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 67 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 67 --jump ACCEPT
--table filter --insert FORWARD --in-interface wlan0 --jump REJECT
--table filter --insert FORWARD --out-interface wlan0 --jump REJECT
--table filter --insert FORWARD --in-interface wlan0 --out-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --source 192.168.1.0/255.255.255.0 --in-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --destination 192.168.1.0/255.255.255.0 --out-interface wlan0 --match state --state ESTABLISHED,RELATED --jump ACCEPT
--table nat --insert POSTROUTING --source 192.168.1.0/255.255.255.0 ! --destination 192.168.1.0/255.255.255.0 --jump MASQUERADE
--version
To me it looks like this way you can change it for specific packages, rather than globally.
The main question you have to ask now is whether you want the change globally or not.
I’ve also tried this with multiple shared connections (after adding dummy interfaces), and it seems NetworkManager properly handles trying to bring them up simultaneously.
I see no different in your outputs, but maybe that is your point.
I think the solutions is good enough. I will just change the script to use iptables
instead of xtables-legacy-multi
as follow:
#!/bin/sh
/usr/sbin/iptables -w 5 "$@"
My networkmanager_%.bbappend look like this:
# Include the iptables retry script
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI_append = " \
file://iptables-retry \
"
# Install iptable retry script and hijack iptables symlink
do_install_append() {
install -m 0755 ${WORKDIR}/iptables-retry ${D}/${sbindir}/iptables-retry
}
EXTRA_OECONF_remove = "--with-iptables=${sbindir}/iptables"
EXTRA_OECONF_append = " --with-iptables=${sbindir}/iptables-retry "
The iptables rules are now ok but I still have problems with my connected wifi clients. They report no internet sometimes but have no problems connecting to my server running in my container.
So how do I troubleshoot whats wrong?
Is there still problems in my routing like with the meter info or iptables rules or what else can it be?
My Huawei phone shows a Hotspot connected symbol when connecting to my Balena router while it show a Wi-fi network connected symbol when connecting to my home wifi router. I don’t understand what differ.
I got a suggestion that is all about metric information. If I get the Hotspot symbol, my phone handle the connection as a not unlimited wifi network.
I looked in the routing table in the begining of this thread and found some metric data. Can that be the problem and how do I fix it?
Wwan0 is the modem and wlan0 is my wifi acting as a Hotspot.
Okey I have found out what make that “Hotspot connected” symbol shown when connecting my phone to my device. It was the Network Manager connection.metered value on my wifi hotspot interface that was set as unknown. By setting the value to “no” will change the symbol to the regular “Wi-Fi network connected”.
I still have some intermittent problems with internet connection and I think the MTU value is the problem. My wwan0 connection (modem) got MTU set to 1430. By change MTU to 1400 on my wifi clients I can get it working.
As I understand the DHCP server can tell wifi clients to use a specific MTU value. How do I set this on my device? I can set 802-11-wireless.mtu
to 1400
but the clients doesn’t care about it.
Got it working by alter the MSS (Maximum Segment Size) value of TCP SYN packets with:
iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -o wwan0 -j TCPMSS --clamp-mss-to-pmtu
So, I guess I have to install iptables in the container and add this rule to it.