Network interfaces route metric

[suporte]

Hi all!

Your product is amazing, although now we are facing a small challenge in networking.

Context:
Our devices run raspberrypi 3 compute module
2 Ethernet NICs
1 4G modem

balenaOs 5.0.8
Supervisor 15.3.0

When launching the app, we configure the network interfaces this way:

---

Eth0

var defaultEthernetManual = InterfaceConfig{
	Address:       "192.168.0.100/24",
	Dhcp:          false,
	DNS:           "192.168.0.1",
	Gateway:       "192.168.0.254",
	Name:          eth0,
	RouteMetric:   "6",
	Type:          "802-3-ethernet",
	InterfaceName: eth0,
}

ipv4.method, manual
ipv4.ignore-auto-dns, yes
ipv4.never-default, true
ipv6.method, disabled

---

Eth1

var defaultEthernetDHCP = InterfaceConfig{
	Dhcp:          true,
	Name:          "",
	RouteMetric:   "4",
	Type:          "802-3-ethernet",
	InterfaceName: "",
}

ipv4.method, auto
ipv4.route-metric, config.RouteMetric
ipv4.dns, 8.8.8.8 193.136.152.81
ipv4.dns-priority, config.RouteMetric
ipv4.ignore-auto-dns, yes
ipv6.method, disabled

---

Cellular

var defaultCellular = InterfaceConfig{
	Dhcp:          true,
	Name:          cellular,
	RouteMetric:   "3",
	Type:          "gsm",
	InterfaceName: "cdc-wdm0",
}

ipv4.method, auto
ipv4.route-metric, config.RouteMetric
ipv4.dns, 8.8.8.8 193.136.152.81
ipv4.dns-priority, config.RouteMetric
ipv4.ignore-auto-dns, yes
ipv6.method, disabled

---

Situation 1

• 4G with antenna connected
• Eth0 and Eth1 connected through a modem with internet access

cdc-wdm0: connected to cellular
“cdc-wdm0”
gsm (option, qmi_wwan), hw, iface wwan0, mtu 1360
ip4 default
inet4 100.75.156.145/30
route4 100.75.156.144/30 metric 3
route4 default via 100.75.156.146 metric 3

supervisor0: connected (externally) to supervisor0
“supervisor0”
bridge, 02:42:B9:0F:BA:6B, sw, mtu 1500
inet4 10.114.104.1/25
route4 10.114.104.0/25 metric 0
inet6 fe80::42:b9ff:fe0f:ba6b/64
route6 fe80::/64 metric 256

eth0: connected to eth0
“eth0”
ethernet (smsc95xx), B8:27:EB:2C:27:83, hw, mtu 1500
inet4 192.168.0.100/24
route4 192.168.0.0/24 metric 6

eth1: connected to eth1
“eth1”
ethernet (smsc95xx), B8:27:EB:64:B7:3A, hw, mtu 1500
inet4 192.168.2.14/24
route4 192.168.2.0/24 metric 4
route4 default via 192.168.2.10 metric 4

Our device prefers the cellular connection as expected.

more info:

cat /etc/resolv.dnsmasq

nameserver 8.8.8.8
nameserver 193.136.152.81
nameserver 192.168.0.1

cat /etc/resolv.conf

we use dnsmasq at 127.0.0.2 so that user containers can run their own dns cache and forwarder and not conflict with dnsmasq on the host

nameserver 127.0.0.2
options timeout:15

nslookup google.com

Server: 127.0.0.2
Address 1: 127.0.0.2 e63bf17
Name: google.com
Address 1: 142.250.185.110 fra16s49-in-f14.1e100.net
Address 2: 2a00:1450:4001:830::200e fra24s11-in-x0e.1e100.net

Situation 2

• took 4G antenna out (lost 4G signal)
• Eth0 and Eth1 linked to a router with internet access

eth1: connected to eth1
“eth1”
ethernet (smsc95xx), B8:27:EB:64:B7:3A, hw, mtu 1500
ip4 default
inet4 192.168.2.14/24
route4 192.168.2.0/24 metric 4
route4 default via 192.168.2.10 metric 4

supervisor0: connected (externally) to supervisor0
“supervisor0”
bridge, 02:42:B9:0F:BA:6B, sw, mtu 1500
inet4 10.114.104.1/25
route4 10.114.104.0/25 metric 0
inet6 fe80::42:b9ff:fe0f:ba6b/64
route6 fe80::/64 metric 256

eth0: connected to eth0
“eth0”
ethernet (smsc95xx), B8:27:EB:2C:27:83, hw, mtu 1500
inet4 192.168.0.100/24
route4 192.168.0.0/24 metric 6

cdc-wdm0: connecting (prepare) to cellular
“cdc-wdm0”
gsm (option, qmi_wwan), hw, mtu 1360

In this case, as expected, it regains internet through the next interface according to the route metric, in this case: Eth1

Situation 3

• I Reconnect the 4G antenna and it regains internet through the cellular network interface

cdc-wdm0: connected to cellular
“cdc-wdm0”
gsm (qmi_wwan, option), hw, iface wwan0, mtu 1360
ip4 default
inet4 100.75.156.145/30
route4 100.75.156.144/30 metric 3
route4 default via 100.75.156.146 metric 3

supervisor0: connected (externally) to supervisor0
“supervisor0”
bridge, 02:42:B9:0F:BA:6B, sw, mtu 1500
inet4 10.114.104.1/25
route4 10.114.104.0/25 metric 0
inet6 fe80::42:b9ff:fe0f:ba6b/64
route6 fe80::/64 metric 256

eth0: connected to eth0
“0424 ec00”
ethernet (smsc95xx), B8:27:EB:2C:27:83, hw, mtu 1500
inet4 192.168.0.100/24
route4 192.168.0.0/24 metric 6

eth1: connected to eth1
“0424 ec00”
ethernet (smsc95xx), B8:27:EB:64:B7:3A, hw, mtu 1500
inet4 192.168.2.14/24
route4 192.168.2.0/24 metric 4
route4 default via 192.168.2.10 metric 4

Situation 4

• I took the 4G antenna (first)
• and then unplug the internet form the router where the Eth0 and Eth1 are connected and of course the device loses internet connection
• note that eth0 and eth1 still have ethernet cables connected to a router (without internet access)

eth1: connected to eth1
“eth1”
ethernet (smsc95xx), B8:27:EB:64:B7:3A, hw, mtu 1500
ip4 default
inet4 192.168.2.14/24
route4 192.168.2.0/24 metric 4
route4 default via 192.168.2.10 metric 4

supervisor0: connected (externally) to supervisor0
“supervisor0”
bridge, 02:42:B9:0F:BA:6B, sw, mtu 1500
inet4 10.114.104.1/25
route4 10.114.104.0/25 metric 0
inet6 fe80::42:b9ff:fe0f:ba6b/64
route6 fe80::/64 metric 256

eth0: connected to eth0
“eth0”
ethernet (smsc95xx), B8:27:EB:2C:27:83, hw, mtu 1500
inet4 192.168.0.100/24
route4 192.168.0.0/24 metric 6

cdc-wdm0: connecting (prepare) to cellular
“cdc-wdm0”
gsm (qmi_wwan, option), hw, mtu 1360

Host
nslookup google.com

Server: 127.0.0.2
Address 1: 127.0.0.2 e63bf17
Name: google.com
Address 1: 10.0.0.1
Address 2: 10.0.0.1

Container
nslookup google.com

Server: 127.0.0.11
Address: 127.0.0.11:53
Name: google.com
Address: 10.0.0.1
Name: google.com
Address: 10.0.0.1

*Situation *

• Finally, I connect the 4G antenna and hope the device to reacquire internet connection through the cellular interface, but alas, it doesn’t!

eth1: connected to eth1
“eth1”
ethernet (smsc95xx), B8:27:EB:64:B7:3A, hw, mtu 1500
ip4 default
inet4 192.168.2.14/24
route4 192.168.2.0/24 metric 4
route4 default via 192.168.2.10 metric 4

supervisor0: connected (externally) to supervisor0
“supervisor0”
bridge, 02:42:B9:0F:BA:6B, sw, mtu 1500
inet4 10.114.104.1/25
route4 10.114.104.0/25 metric 0
inet6 fe80::42:b9ff:fe0f:ba6b/64
route6 fe80::/64 metric 256

eth0: connected to eth0
“eth0”
ethernet (smsc95xx), B8:27:EB:2C:27:83, hw, mtu 1500
inet4 192.168.0.100/24
route4 192.168.0.0/24 metric 6

cdc-wdm0: connected to cellular
“cdc-wdm0”
gsm (qmi_wwan, option), hw, iface wwan0, mtu 1360
inet4 100.75.156.145/30
route4 100.75.156.144/30 metric 3
route4 default via 100.75.156.146 metric 20003

---

Status | lock: sim-pin2
| unlock retries: sim-pin (3), sim-puk (10), sim-pin2 (3), sim-puk2 (10)
| state: connected
| power state: on
| access tech: lte
| signal quality: 60% (recent)

It recognizes and connects to the 4G signal but doesn’t use that interface. It somehow “invented” the route metric 20003.

Any idea for how does this happen and how to avoid it?
Thank you in advance and best regards

[mpous]

Hello @suporte it’s really odd that the route4 default has the metric re-written!

I’m checking the nmcli forums to explore if there is a case there like yours!

Let’s stay connected if you find a solution on this!

[suporte]

Hi @mpous

Thanks for the quick response.

You might find our previous reported issue in this post

BalenaOS DNS failure

Hi, glad to hear it’s working If you don’t want to discard the DHCP DNS altogether, it seems there’s also a setting ipv4.dns-priority that might help in fixing the order of your nameservers. Maybe try setting that on both connections, with the cellular having a lower value; note that the default values are 50 for VPN and 100 for others and 0 selects the default. Based on your routing table, it should then first try the 172.30.8.5 and 172.30.8.6 via cdc-wdm0, only going to 192.…

And perhaps I should have continued that, instead of creating a new one.

We’re already implementing the suggestions made in that post but something is still missing…

Best regards!

[suporte]

If anyone has some hint on this, I would appreciate a lot.

If someone on the Balena Team’s side needs, we can give access to a device and I can perform the tests (plug cable, unplug cable,…) at the same time

All the best for everyone!