So it might be balena-engine that conflicts with Network manager. I guess there is not much to do about.
I hope balena-engine use that retry flag when adding iptables rules so it can be sure that rules are added.
So what is best practice to run my fix script? Should it be activated by some kind of event like balena firewall rules is added or should I just let the script run at boot and check iptables if rules are added and then fix what is missing?
Here is my script. I guess it can be improved so feel free to replay modifications.
#!/bin/bash
while ! iptables -n --list "BALENA-FIREWALL" > /dev/null 2>&1; do
echo "Wating for Balena firewall rules to be set...."
sleep 5s
done
if ! iptables -C INPUT -i wlan0 -p udp -m udp --dport 67 -j ACCEPT; then
echo "Adding: INPUT -i wlan0 -p udp -m udp --dport 67 -j ACCEPT"
iptables -A INPUT -i wlan0 -p udp -m udp --dport 67 -j ACCEPT -w 5
fi
if ! iptables -C INPUT -i wlan0 -p tcp -m tcp --dport 67 -j ACCEPT; then
echo "Adding: INPUT -i wlan0 -p tcp -m tcp --dport 67 -j ACCEPT"
iptables -A INPUT -i wlan0 -p tcp -m tcp --dport 67 -j ACCEPT -w 5
fi
if ! iptables -C INPUT -i wlan0 -p udp -m udp --dport 53 -j ACCEPT; then
echo "Adding: INPUT -i wlan0 -p udp -m udp --dport 53 -j ACCEPT"
iptables -A INPUT -i wlan0 -p udp -m udp --dport 53 -j ACCEPT -w 5
fi
if ! iptables -C INPUT -i wlan0 -p tcp -m tcp --dport 53 -j ACCEPT; then
echo "Adding: INPUT -i wlan0 -p tcp -m tcp --dport 53 -j ACCEPT"
iptables -A INPUT -i wlan0 -p tcp -m tcp --dport 53 -j ACCEPT -w 5
fi
if ! iptables -C FORWARD -d 192.168.1.0/24 -o wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT; then
echo "Adding: FORWARD -d 192.168.1.0/24 -o wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT"
iptables -A FORWARD -d 192.168.1.0/24 -o wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT -w 5
fi
if ! iptables -C FORWARD -s 192.168.1.0/24 -i wlan0 -j ACCEPT; then
echo "Adding: FORWARD -s 192.168.1.0/24 -i wlan0 -j ACCEPT"
iptables -A FORWARD -s 192.168.1.0/24 -i wlan0 -j ACCEPT -w 5
fi
if ! iptables -C FORWARD -i wlan0 -o wlan0 -j ACCEPT; then
echo "Adding: FORWARD -i wlan0 -o wlan0 -j ACCEPT"
iptables -A FORWARD -i wlan0 -o wlan0 -j ACCEPT -w 5
fi
if ! iptables -C FORWARD -o wlan0 -j REJECT --reject-with icmp-port-unreachable; then
echo "Adding: FORWARD -o wlan0 -j REJECT --reject-with icmp-port-unreachable"
iptables -A FORWARD -o wlan0 -j REJECT --reject-with icmp-port-unreachable -w 5
fi
if ! iptables -C FORWARD -i wlan0 -j REJECT --reject-with icmp-port-unreachable; then
echo "Adding: FORWARD -i wlan0 -j REJECT --reject-with icmp-port-unreachable"
iptables -A FORWARD -i wlan0 -j REJECT --reject-with icmp-port-unreachable -w 5
fi
Below is the output from the script. Why do I get these bad rule warnings? The rules are added but it complains.
Wating for Balena firewall rules to be set....
Wating for Balena firewall rules to be set....
Wating for Balena firewall rules to be set....
Wating for Balena firewall rules to be set....
iptables: Bad rule (does a matching rule exist in that chain?).
Adding: INPUT -i wlan0 -p udp -m udp --dport 67 -j ACCEPT
iptables: Bad rule (does a matching rule exist in that chain?).
Adding: INPUT -i wlan0 -p tcp -m tcp --dport 67 -j ACCEPT
iptables: Bad rule (does a matching rule exist in that chain?).
Adding: INPUT -i wlan0 -p udp -m udp --dport 53 -j ACCEPT
iptables: Bad rule (does a matching rule exist in that chain?).
Adding: INPUT -i wlan0 -p tcp -m tcp --dport 53 -j ACCEPT
iptables: Bad rule (does a matching rule exist in that chain?).
Adding: FORWARD -i wlan0 -o wlan0 -j ACCEPT
iptables: No chain/target/match by that name.
Adding: FORWARD -o wlan0 -j REJECT --reject-with icmp-port-unreachable
Would you mind sharing your docker setup so I can try out a few things this weekend?
I use this layer with some smaller changes: https://github.com/srednak/balena-st-stm32mp
You can find my networkmanager connection configuration above in this thread.
The problem is that I don’t think you can replicate the collision I got as it was gone after I added my script-file and service that will start the script without enabling it. Balena firewall still remove those INPUT rules, but internet sharing is working (If you use other DNS server than default 192.168.1.1.
It’s only if networkmanager can’t create those FORWARD rules, internet sharing will not work.
Hi,
I played around a bit today.
It was quite easy to replicate NetworkManager not being able to add the routes.
Just need a shared interface and a terminal in the HostOS.
flock /run/xtables.lock -c "sleep 120s")nmcli con up balena-hotspot)I’ve also tried this with multiple shared connections (after adding dummy interfaces), and it seems NetworkManager properly handles trying to bring them up simultaneously.
I’ve had a look at NetworkManager dispatcher scripts, but you can’t halt the main process from those because they are spawned in a separate process. So while your script there may wait for the lockfile to become accessible (and prevent the up command from returning), the iptables commands will still run (and fail) before your script finishes.
For the balena-enginepart, I tried balena network create test, which adds some rules to the nat table.
It was indeed quite difficult to get this to occur at the same time as the NetworkManager bit.
However, running the command while holding the lockfile, showed the balena command waits properly for the lock to be released before adding the rules.
It seems like you were able to consistently recreate the balena/NetworkManager timing.
So something I’m curious about is if you can create a wrapper script for iptables in the Host OS to see if the wait flag actually fixes this.
For example, you could replace the /usr/sbin/iptables symbolic link with the following script:
#!/bin/sh
/usr/sbin/xtables-legacy-multi iptables -w 5 "$@"
Thank you very much. I tried that trick with hijacking iptables symbol link, and it worked every time.
I have restart the devices multiple times. Now I only have to implement the hijack in my recipe.
When iptables look like this it usually does not work as it competes with balena-engine to add rules.
-A INPUT -j BALENA-FIREWALL
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o balena0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o balena0 -j DOCKER
-A FORWARD -i balena0 ! -o balena0 -j ACCEPT
-A FORWARD -i balena0 -o balena0 -j ACCEPT
-A FORWARD -d 192.168.1.0/24 -o wlan0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -s 192.168.1.0/24 -i wlan0 -j ACCEPT
-A FORWARD -i wlan0 -o wlan0 -j ACCEPT
-A FORWARD -o supervisor0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o supervisor0 -j DOCKER
-A FORWARD -i supervisor0 ! -o supervisor0 -j ACCEPT
-A FORWARD -i supervisor0 -o supervisor0 -j ACCEPT
-A FORWARD -o br-a8483e87e992 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-a8483e87e992 -j DOCKER
-A FORWARD -o wlan0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i wlan0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i br-a8483e87e992 ! -o br-a8483e87e992 -j ACCEPT
-A FORWARD -i br-a8483e87e992 -o br-a8483e87e992 -j ACCEPT
-A BALENA-FIREWALL -m state --state RELATED,ESTABLISHED -j ACCEPT
-A BALENA-FIREWALL -m addrtype --src-type LOCAL -j ACCEPT
-A BALENA-FIREWALL -i resin-vpn -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i tun0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i docker0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i lo -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i supervisor0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -p tcp -m tcp --dport 48484 -j REJECT --reject-with icmp-port-unreachable
-A BALENA-FIREWALL -p tcp -m tcp --dport 22222 -j ACCEPT
-A BALENA-FIREWALL -p tcp -m tcp --dport 2375 -j ACCEPT
-A BALENA-FIREWALL -m addrtype --dst-type MULTICAST -j ACCEPT
-A BALENA-FIREWALL -p icmp -j ACCEPT
-A BALENA-FIREWALL -i balena0 -p udp -m udp --dport 53 -j ACCEPT
-A BALENA-FIREWALL -j RETURN
-A BALENA-FIREWALL -j REJECT --reject-with icmp-port-unreachable
-A DOCKER-ISOLATION-STAGE-1 -i balena0 ! -o balena0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i supervisor0 ! -o supervisor0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-a8483e87e992 ! -o br-a8483e87e992 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o balena0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o supervisor0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-a8483e87e992 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
Where is best place to put this modification in my Yocto layer?
I created a iptables_%.bbappend file and added following code:
# Include the iptables retry script
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI_append = " \
file://iptables-retry \
"
# Install iptable retry script and hijack iptables symlink
do_install_append() {
install -m 0755 ${WORKDIR}/iptables-retry ${D}/${sbindir}/iptables-retry
ln -sf iptables-retry ${D}/${sbindir}/iptables
}
The script is like TjVv suggested:
#!/bin/sh
/usr/sbin/xtables-legacy-multi iptables -w 5 "$@"
The above solution is working but maybe there is a more neat way to do it.
What about changing
–with-iptables=${sbindir}/iptables
in the networkmanager recipie? Is it possible to add the -w 5 argument here?
I’m not quite sure about the place for to inject this workaround; will poke around tonight a bit.
I will say that before using this in the wild, the script will need a little input checking, for example to prevent a second -w or --wait argument.
As we know networkmanager does not use --wait or -w we can put the installation on this workaround script in a bbappend for networkmanager instead and just change –with-iptables
In that way the workaround will only affect networkmanager.
Does supervisor container have iptables installed and run same command at same time?
How do I check that?
Life (including work) got in the way of looking at the yocto recipes.
Regarding iptables in supervisor, you can grab the sources from github and search through it.
You will see iptables is installed and used to allow access to VPN and supervisor.
I’ve had a look at injecting this and had some success it seems.
Based on your work, I made a recipes-connectivity/networkmanager/networkmanager_%.bbappend:
# Include the iptables retry script
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI_append = " \
file://fake-iptables.sh \
"
# Install iptable retry script and hijack iptables symlink
do_install_append() {
install -m 0755 ${WORKDIR}/fake-iptables.sh ${D}/${sbindir}/fake-iptables
}
EXTRA_OECONF += " --with-iptables=${sbindir}/fake-iptables"
The script it refers to is recipes-connectivity/networkmanager/files/fake-iptables.sh:
#!/bin/sh
echo "$@" >> /tmp/iptables.log
/usr/sbin/iptables "$@"
Building and flashing the image, shows me the following:
iptables works normally as expectedshared connection in NetworkManager, the relevant rules pop up in the log file I use in the fake-iptables script.root@86a5d95:~# cat /tmp/iptables.log
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 67 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 67 --jump ACCEPT
--table filter --insert FORWARD --in-interface wlan0 --jump REJECT
--table filter --insert FORWARD --out-interface wlan0 --jump REJECT
--table filter --insert FORWARD --in-interface wlan0 --out-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --source 192.168.1.0/255.255.255.0 --in-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --destination 192.168.1.0/255.255.255.0 --out-interface wlan0 --match state --state ESTABLISHED,RELATED --jump ACCEPT
--table nat --insert POSTROUTING --source 192.168.1.0/255.255.255.0 ! --destination 192.168.1.0/255.255.255.0 --jump MASQUERADE
Here you can see the normal iptables vs fake-iptables operation:
root@86a5d95:~# iptables --version
iptables v1.8.4 (legacy)
root@86a5d95:~# cat /tmp/iptables.log
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 67 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 67 --jump ACCEPT
--table filter --insert FORWARD --in-interface wlan0 --jump REJECT
--table filter --insert FORWARD --out-interface wlan0 --jump REJECT
--table filter --insert FORWARD --in-interface wlan0 --out-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --source 192.168.1.0/255.255.255.0 --in-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --destination 192.168.1.0/255.255.255.0 --out-interface wlan0 --match state --state ESTABLISHED,RELATED --jump ACCEPT
--table nat --insert POSTROUTING --source 192.168.1.0/255.255.255.0 ! --destination 192.168.1.0/255.255.255.0 --jump MASQUERADE
root@86a5d95:~# fake-iptables --version
iptables v1.8.4 (legacy)
root@86a5d95:~# cat /tmp/iptables.log
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 53 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol tcp --destination-port 67 --jump ACCEPT
--table filter --insert INPUT --in-interface wlan0 --protocol udp --destination-port 67 --jump ACCEPT
--table filter --insert FORWARD --in-interface wlan0 --jump REJECT
--table filter --insert FORWARD --out-interface wlan0 --jump REJECT
--table filter --insert FORWARD --in-interface wlan0 --out-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --source 192.168.1.0/255.255.255.0 --in-interface wlan0 --jump ACCEPT
--table filter --insert FORWARD --destination 192.168.1.0/255.255.255.0 --out-interface wlan0 --match state --state ESTABLISHED,RELATED --jump ACCEPT
--table nat --insert POSTROUTING --source 192.168.1.0/255.255.255.0 ! --destination 192.168.1.0/255.255.255.0 --jump MASQUERADE
--version
To me it looks like this way you can change it for specific packages, rather than globally.
The main question you have to ask now is whether you want the change globally or not.
I’ve also tried this with multiple shared connections (after adding dummy interfaces), and it seems NetworkManager properly handles trying to bring them up simultaneously.
I see no different in your outputs, but maybe that is your point.
I think the solutions is good enough. I will just change the script to use iptables instead of xtables-legacy-multi as follow:
#!/bin/sh
/usr/sbin/iptables -w 5 "$@"
My networkmanager_%.bbappend look like this:
# Include the iptables retry script
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI_append = " \
file://iptables-retry \
"
# Install iptable retry script and hijack iptables symlink
do_install_append() {
install -m 0755 ${WORKDIR}/iptables-retry ${D}/${sbindir}/iptables-retry
}
EXTRA_OECONF_remove = "--with-iptables=${sbindir}/iptables"
EXTRA_OECONF_append = " --with-iptables=${sbindir}/iptables-retry "
The iptables rules are now ok but I still have problems with my connected wifi clients. They report no internet sometimes but have no problems connecting to my server running in my container.
So how do I troubleshoot whats wrong?
Is there still problems in my routing like with the meter info or iptables rules or what else can it be?
My Huawei phone shows a Hotspot connected symbol when connecting to my Balena router while it show a Wi-fi network connected symbol when connecting to my home wifi router. I don’t understand what differ.
I got a suggestion that is all about metric information. If I get the Hotspot symbol, my phone handle the connection as a not unlimited wifi network.
I looked in the routing table in the begining of this thread and found some metric data. Can that be the problem and how do I fix it?
Wwan0 is the modem and wlan0 is my wifi acting as a Hotspot.
