iptables rules set from Tailscale are overwritten by Balena?

ada · March 12, 2024, 5:05pm

Hi there

We are using Tailscale to connect our Balena devices together in a VPN. This works great. Recently we started using the Tailscale subnet router functionality. This exposes an IP subnet only accessible by a specific Balena device, and allows other devices inside the Tailscale VPN to access those IP addresses directly. Tailscale is installed as service, privileged and with network host mode.

The issue we found is, it works, but only after restarting the Tailscale service once the device is ready. It looks like it either could not add the iptables entries yet or they disappeared.

After boot:

root@0d1bf61:~# iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N BALENA-FIREWALL
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N ts-forward
-N ts-input
-A INPUT -j BALENA-FIREWALL
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o balena0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o balena0 -j DOCKER
-A FORWARD -i balena0 ! -o balena0 -j ACCEPT
-A FORWARD -i balena0 -o balena0 -j ACCEPT
-A FORWARD -o supervisor0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o supervisor0 -j DOCKER
-A FORWARD -i supervisor0 ! -o supervisor0 -j ACCEPT
-A FORWARD -i supervisor0 -o supervisor0 -j ACCEPT
-A FORWARD -o br-e52f2f65ceac -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-e52f2f65ceac -j DOCKER
-A FORWARD -i br-e52f2f65ceac ! -o br-e52f2f65ceac -j ACCEPT
-A FORWARD -i br-e52f2f65ceac -o br-e52f2f65ceac -j ACCEPT
-A BALENA-FIREWALL -m state --state RELATED,ESTABLISHED -j ACCEPT
-A BALENA-FIREWALL -m addrtype --src-type LOCAL -j ACCEPT
-A BALENA-FIREWALL -i resin-vpn -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i tun0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i docker0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i lo -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i supervisor0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -p tcp -m tcp --dport 48484 -j REJECT --reject-with icmp-port-unreachable
-A BALENA-FIREWALL -p tcp -m tcp --dport 22222 -j ACCEPT
-A BALENA-FIREWALL -p tcp -m tcp --dport 2375 -j ACCEPT
-A BALENA-FIREWALL -m addrtype --dst-type MULTICAST -j ACCEPT
-A BALENA-FIREWALL -p icmp -j ACCEPT
-A BALENA-FIREWALL -i balena0 -p udp -m udp --dport 53 -j ACCEPT
-A BALENA-FIREWALL -j RETURN
-A BALENA-FIREWALL -j REJECT --reject-with icmp-port-unreachable
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14555 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14554 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14553 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14552 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14551 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14550 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14510 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14509 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14508 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14507 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14506 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14505 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14504 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14503 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14502 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14501 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14500 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p tcp -m tcp --dport 8181 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i balena0 ! -o balena0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i supervisor0 ! -o supervisor0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-e52f2f65ceac ! -o br-e52f2f65ceac -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o balena0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o supervisor0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-e52f2f65ceac -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN

After restarting Tailscale again:

root@0d1bf61:~# iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N BALENA-FIREWALL
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N ts-forward
-N ts-input
-A INPUT -j ts-input
-A INPUT -j BALENA-FIREWALL
-A FORWARD -j ts-forward
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o balena0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o balena0 -j DOCKER
-A FORWARD -i balena0 ! -o balena0 -j ACCEPT
-A FORWARD -i balena0 -o balena0 -j ACCEPT
-A FORWARD -o supervisor0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o supervisor0 -j DOCKER
-A FORWARD -i supervisor0 ! -o supervisor0 -j ACCEPT
-A FORWARD -i supervisor0 -o supervisor0 -j ACCEPT
-A FORWARD -o br-e52f2f65ceac -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-e52f2f65ceac -j DOCKER
-A FORWARD -i br-e52f2f65ceac ! -o br-e52f2f65ceac -j ACCEPT
-A FORWARD -i br-e52f2f65ceac -o br-e52f2f65ceac -j ACCEPT
-A BALENA-FIREWALL -m state --state RELATED,ESTABLISHED -j ACCEPT
-A BALENA-FIREWALL -m addrtype --src-type LOCAL -j ACCEPT
-A BALENA-FIREWALL -i resin-vpn -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i tun0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i docker0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i lo -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -i supervisor0 -p tcp -m tcp --dport 48484 -j ACCEPT
-A BALENA-FIREWALL -p tcp -m tcp --dport 48484 -j REJECT --reject-with icmp-port-unreachable
-A BALENA-FIREWALL -p tcp -m tcp --dport 22222 -j ACCEPT
-A BALENA-FIREWALL -p tcp -m tcp --dport 2375 -j ACCEPT
-A BALENA-FIREWALL -m addrtype --dst-type MULTICAST -j ACCEPT
-A BALENA-FIREWALL -p icmp -j ACCEPT
-A BALENA-FIREWALL -i balena0 -p udp -m udp --dport 53 -j ACCEPT
-A BALENA-FIREWALL -j RETURN
-A BALENA-FIREWALL -j REJECT --reject-with icmp-port-unreachable
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14555 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14554 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14553 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14552 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14551 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14550 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14510 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14509 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14508 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14507 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14506 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14505 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14504 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14503 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14502 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14501 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p udp -m udp --dport 14500 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i br-e52f2f65ceac -o br-e52f2f65ceac -p tcp -m tcp --dport 8181 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i balena0 ! -o balena0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i supervisor0 ! -o supervisor0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-e52f2f65ceac ! -o br-e52f2f65ceac -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o balena0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o supervisor0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-e52f2f65ceac -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A ts-forward -i tailscale0 -j MARK --set-xmark 0x40000/0xff0000
-A ts-forward -m mark --mark 0x40000/0xff0000 -j ACCEPT
-A ts-forward -s 100.64.0.0/10 -o tailscale0 -j DROP
-A ts-forward -o tailscale0 -j ACCEPT
-A ts-input -s 100.76.39.10/32 -i lo -j ACCEPT
-A ts-input -s 100.115.92.0/23 ! -i tailscale0 -j RETURN
-A ts-input -s 100.64.0.0/10 ! -i tailscale0 -j DROP
-A ts-input -i tailscale0 -j ACCEPT
-A ts-input -p udp -m udp --dport 52583 -j ACCEPT

You notice the added rules in the end. This problem exists on some very old OS versions (2.x) but also on the latest 5.x.

Is this a startup order issue or task that cleans up rules after all services have started? Looks like we have to make it wait somehow to re-configure iptables after some time.

Regards,

ada · May 8, 2024, 12:51pm

If someone else has the same or similar issue, our workaround is to let Tailscale start last and use a sleep time before starting it. This seems to work consistently.

  tailscale:
    image: tailscale-image
    # Tailscale needs to start delayed to be able to fix iptables after Balena has booted.
    command: sh -c "
      sleep 30;
      if [ -z $TAILSCALE_KEY ]; then
        echo 'Set TAILSCALE_KEY to activate.';
        sleep infinity;
      else
        (sleep 10 &&
          tailscale up --accept-routes --advertise-routes=$TAILSCALE_SUBNETS --authkey=$TAILSCALE_KEY --hostname=$BALENA_DEVICE_NAME_AT_INIT
        ) & tailscaled --state /data/tailscaled.state;
      fi"
    restart: always
    network_mode: host
    # The hope is that balena made the changes after all servics started, including this one,
    # and with the 30s delay the tailscale stuff gets applied last consistently.
    depends_on:
      - everyOtherContainer
    privileged: true
    cap_add:
      - NET_ADMIN
    volumes:
      - 'data-volume:/data'
    devices:
      - /dev/net/tun:/dev/net/tun

Topic		Replies	Views
Balena application iptables no longer propagating to host BalenaOS balenaOS network	4	939	December 10, 2020
Help with building a remote tunnle Project help	1	75	May 22, 2024
balenaPihole and Tailscale VPN Project help	1	264	January 12, 2024
Help needed: Persist iptables Rule on the Host OS [Network Requirements] balenaOS network , pendinguserresponse	11	2341	May 8, 2019
UDEV rules and forwarding rules balenaOS	7	2150	December 3, 2019

iptables rules set from Tailscale are overwritten by Balena?

Related topics