Networking in balenaOS with Cellular & Ethernet

I have a Raspberry Pi 4 with a SixFab LTE HAT, with the EG25-G Cellular modem with a T-Mobile SIM. Originally I had a (very expensive) Sixfab Core SIM in it and decided to move to a prepaid T-Mobile data plan.

With regards to networking, I notice on Sixfab Core and T-Mobile SIMs, interface enp1s0u1u3i4 has a private IP address on the 192.168.255.x/24 network and, with Sixfab, had the highest metric for default route out of the box. When I initiated a ping -I enp1s0u1u3i4 -c 5 google.com the traffic would flow through that interface out of cellular.

When I switched to the T-Mobile SIM the priority went to eth0, enp1s0u1u3i4 and then ppp0 (which has the public IP address from T-Mobile). When I run a ping -I enp1s0u1u3i4 -c google.com it fails. When I run a ping -I ppp0 -c google.com it works. NOTE: I did re-image BalenaOS and put a fresh cellular config in the system-connections folder.

Here is my current default route table:

root@e7d5af5:~# ip route
default via 192.168.7.1 dev eth0  metric 100 
default via 192.168.225.1 dev enp1s0u1u3i4  metric 20101 
default dev ppp0 scope link  metric 20700 

Here is my network configuration for cellular:

[connection]
id=T-Mobile
type=gsm
autoconnect=true

[gsm]
apn=fast.t-mobile.com

[serial]
baud=115200

[ipv4]
method=auto

[ipv6]
addr-gen-mode=stable-privacy
method=auto

Here is NetworkManager showing the connection as active (both are dark green and imply good connection):

root@e7d5af5:~# nmcli c
NAME                UUID                                  TYPE      DEVICE      
Wired connection 2  4e88fed9-82fd-3e5f-90a6-44bfb1db91bb  ethernet  eth0        
T-Mobile            a937294d-10a3-3ec5-b0e7-a588eddce766  gsm       ttyUSB2  

Modem Manager:

root@e7d5af5:~# mmcli -m 0
  --------------------------------
  General  |                 path: /org/freedesktop/ModemManager1/Modem/0
           |            device id: <redacted>
  --------------------------------
  Hardware |         manufacturer: Quectel
           |                model: EG25
           |    firmware revision: EG25GGBR07A08M2G
           |            supported: gsm-umts, lte
           |              current: gsm-umts, lte
           |         equipment id: <redacted>
  --------------------------------
  System   |               device: /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.3
           |              drivers: option, cdc_ether
           |               plugin: quectel
           |         primary port: ttyUSB2
           |                ports: enp1s0u1u3i4 (net), ttyUSB0 (qcdm), ttyUSB1 (gps), 
           |                       ttyUSB2 (at), ttyUSB3 (at)
  --------------------------------
  Numbers  |                  own: <redacted>
  --------------------------------
  Status   |       unlock retries: sim-pin (3), sim-puk (10), sim-pin2 (10), sim-puk2 (10)
           |                state: connected
           |          power state: on
           |          access tech: lte
           |       signal quality: 40% (recent)
  --------------------------------
  Modes    |            supported: allowed: 2g, 3g, 4g; preferred: none
           |              current: allowed: 2g, 3g, 4g; preferred: none
  --------------------------------
  IP       |            supported: ipv4, ipv6, ipv4v6
  --------------------------------
  3GPP     |                 imei: <redacted>
           |          operator id: <redacted>
           |        operator name: T-Mobile
           |         registration: home
  --------------------------------
  3GPP EPS | ue mode of operation: csps-1
  --------------------------------
  SIM      |     primary sim path: /org/freedesktop/ModemManager1/SIM/0
  --------------------------------
  Bearer   |                paths: /org/freedesktop/ModemManager1/Bearer/0
root@e7d5af5:~# journalctl -u ModemManager
-- Journal begins at Sat 2022-07-09 21:02:04 UTC, ends at Sat 2022-07-09 21:15:00 UTC. --
Jul 09 21:02:07 localhost ModemManager[1476]: ModemManager (version 1.18.4) starting in system bus...
Jul 09 21:02:19 e7d5af5 ModemManager[1476]: [device /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00>
Jul 09 21:02:19 e7d5af5 ModemManager[1476]: [base-manager] modem for device '/sys/devices/platform/scb/fd500000.pcie/pci0000:0>
Jul 09 21:02:30 e7d5af5 ModemManager[1476]: [modem0/sim0] couldn't load operator name: Failed to parse CRSM query result '+CRS>
Jul 09 21:02:30 e7d5af5 ModemManager[1476]: [modem0/sim0] couldn't load list of preferred networks: Operation not allowed
Jul 09 21:02:59 e7d5af5 ModemManager[1476]: [modem0/ttyUSB0/qcdm] close blocked by driver for more than 7 seconds!
Jul 09 21:03:00 e7d5af5 ModemManager[1476]: [modem0] state changed (unknown -> disabled)
Jul 09 21:03:00 e7d5af5 ModemManager[1476]: [modem0] state changed (disabled -> enabling)
Jul 09 21:03:05 e7d5af5 ModemManager[1476]: [modem0] simple connect started...
Jul 09 21:03:05 e7d5af5 ModemManager[1476]: [modem0] simple connect state (4/8): wait to get fully enabled
Jul 09 21:03:05 e7d5af5 ModemManager[1476]: [modem0] power state updated: on
Jul 09 21:03:06 e7d5af5 ModemManager[1476]: [modem0] 3GPP registration state changed (unknown -> idle)
Jul 09 21:03:06 e7d5af5 ModemManager[1476]: [modem0] state changed (enabling -> enabled)
Jul 09 21:03:06 e7d5af5 ModemManager[1476]: [modem0] simple connect state (5/8): register
Jul 09 21:03:06 e7d5af5 ModemManager[1476]: [modem0] 3GPP registration state changed (idle -> registering)
Jul 09 21:03:06 e7d5af5 ModemManager[1476]: [modem0] 3GPP registration state changed (registering -> home)
Jul 09 21:03:06 e7d5af5 ModemManager[1476]: [modem0] state changed (enabled -> registered)
Jul 09 21:03:06 e7d5af5 ModemManager[1476]: [modem0] simple connect state (6/8): bearer
Jul 09 21:03:06 e7d5af5 ModemManager[1476]: [modem0] simple connect state (7/8): connect
Jul 09 21:03:06 e7d5af5 ModemManager[1476]: [modem0] state changed (registered -> connecting)
Jul 09 21:03:06 e7d5af5 ModemManager[1476]: [modem0] state changed (connecting -> connected)
Jul 09 21:03:06 e7d5af5 ModemManager[1476]: [modem0] simple connect state (8/8): all done
root@e7d5af5:~# 
root@e7d5af5:~# journalctl -u NetworkManager
-- Journal begins at Sat 2022-07-09 21:02:04 UTC, ends at Sat 2022-07-09 21:16:17 UTC. --
Jul 09 21:02:08 localhost NetworkManager[1554]: <info>  [1657400528.0814] NetworkManager (version 1.32.12) is starting... (for>
Jul 09 21:02:08 localhost NetworkManager[1554]: <info>  [1657400528.0818] Read config: /etc/NetworkManager/NetworkManager.conf>
Jul 09 21:02:08 localhost NetworkManager[1554]: <info>  [1657400528.1039] bus-manager: acquired D-Bus service "org.freedesktop>
Jul 09 21:02:08 localhost NetworkManager[1554]: <info>  [1657400528.1884] manager[0x556e55f090]: monitoring kernel firmware di>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.2506] hostname: hostname: using hostnamed
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.2510] hostname: hostname changed from (none) to "e7d5af5"
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.2554] dns-mgr[0x556e53f250]: init: dns=default,systemd-resol>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.2604] rfkill0: found Wi-Fi radio killswitch (at /sys/devices>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.2628] manager[0x556e55f090]: rfkill: Wi-Fi hardware radio se>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.2634] manager[0x556e55f090]: rfkill: WWAN hardware radio set>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3168] Loaded device plugin: NMBluezManager (/usr/lib/Network>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3208] Loaded device plugin: NMAtmManager (/usr/lib/NetworkMa>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3279] Loaded device plugin: NMWifiFactory (/usr/lib/NetworkM>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3317] Loaded device plugin: NMWwanFactory (/usr/lib/NetworkM>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3327] manager: rfkill: Wi-Fi enabled by radio killswitch; en>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3336] manager: rfkill: WWAN enabled by radio killswitch; ena>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3339] manager: Networking is enabled by state file
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3346] dhcp-init: Using DHCP client 'internal'
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3392] settings: Loaded settings plugin: keyfile (internal)
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3577] device (lo): carrier: link connected
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3587] manager: (lo): new Generic device (/org/freedesktop/Ne>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3764] manager: (enp1s0u1u3i4): new Ethernet device (/org/fre>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3906] settings: (enp1s0u1u3i4): created default wired connec>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.3952] device (enp1s0u1u3i4): state change: unmanaged -> unav>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.4006] device (enp1s0u1u3i4): carrier: link connected
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.4092] manager: (eth0): new Ethernet device (/org/freedesktop>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.4271] settings: (eth0): created default wired connection 'Wi>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.4292] device (eth0): state change: unmanaged -> unavailable >
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.4557] manager: (resin-dns): new Bridge device (/org/freedesk>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.4676] device (wlan0): driver supports Access Point (AP) mode
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.4714] manager: (wlan0): new 802.11 Wi-Fi device (/org/freede>
Jul 09 21:02:09 e7d5af5 NetworkManager[1554]: <info>  [1657400529.4948] device (wlan0): state change: unmanaged -> unavailable>
Jul 09 21:02:10 e7d5af5 NetworkManager[1554]: <info>  [1657400530.0469] device (wlan0): set-hw-addr: set MAC address to AA:E2:>
Jul 09 21:02:10 e7d5af5 NetworkManager[1554]: <info>  [1657400530.1092] device (enp1s0u1u3i4): state change: unavailable -> di>
Jul 09 21:02:10 e7d5af5 NetworkManager[1554]: <info>  [1657400530.1147] policy: auto-activating connection 'Wired connection 1>
Jul 09 21:02:10 e7d5af5 NetworkManager[1554]: <info>  [1657400530.1263] device (enp1s0u1u3i4): Activation: starting connection>
Jul 09 21:02:10 e7d5af5 NetworkManager[1554]: <info>  [1657400530.1270] device (enp1s0u1u3i4): state change: disconnected -> p>
Jul 09 21:02:10 e7d5af5 NetworkManager[1554]: <info>  [1657400530.1339] manager: NetworkManager state is now CONNECTING
Jul 09 21:02:10 e7d5af5 NetworkManager[1554]: <info>  [1657400530.1355] device (enp1s0u1u3i4): state change: prepare -> config>
Jul 09 21:02:10 e7d5af5 NetworkManager[1554]: <info>  [1657400530.1398] device (enp1s0u1u3i4): state change: config -> ip-conf>
Jul 09 21:02:10 e7d5af5 NetworkManager[1554]: <info>  [1657400530.1480] dhcp4 (enp1s0u1u3i4): activation: beginning transactio>
Jul 09 21:02:10 e7d5af5 NetworkManager[1554]: <info>  [1657400530.2906] device (wlan0): supplicant interface state: internal-s>
Jul 09 21:02:10 e7d5af5 NetworkManager[1554]: <info>  [1657400530.2911] device (wlan0): state change: unavailable -> disconnec>
Jul 09 21:02:11 e7d5af5 NetworkManager[1554]: <info>  [1657400531.6554] manager: (balena0): new Bridge device (/org/freedeskto>
Jul 09 21:02:11 e7d5af5 NetworkManager[1554]: <info>  [1657400531.8768] manager: (supervisor0): new Bridge device (/org/freede>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.0931] device (supervisor0): state change: unmanaged -> unava>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <warn>  [1657400532.1111] device (supervisor0): failed to read bridge setting 'v>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <warn>  [1657400532.1115] device (supervisor0): failed to read bridge setting 'v>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.1234] device (supervisor0): state change: unavailable -> dis>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.1293] device (supervisor0): Activation: starting connection >
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.1339] manager: (br-98d2b6378cb4): new Bridge device (/org/fr>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.1488] device (supervisor0): state change: disconnected -> pr>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.1515] device (supervisor0): state change: prepare -> config >
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.1558] device (supervisor0): state change: config -> ip-confi>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.1586] device (supervisor0): state change: ip-config -> ip-ch>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.1726] device (supervisor0): state change: ip-check -> second>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.1837] device (supervisor0): state change: secondaries -> act>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.1877] device (supervisor0): Activation: successful, device a>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.9741] manager: (vethfb615ad): new Veth device (/org/freedesk>
Jul 09 21:02:12 e7d5af5 NetworkManager[1554]: <info>  [1657400532.9860] manager: (veth7eb0370): new Veth device (/org/freedesk>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.1950] dhcp4 (enp1s0u1u3i4): state changed unknown -> bound, >
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.2031] device (enp1s0u1u3i4): state change: ip-config -> ip-c>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.2227] device (enp1s0u1u3i4): state change: ip-check -> secon>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.2338] device (enp1s0u1u3i4): state change: secondaries -> ac>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.2393] manager: NetworkManager state is now CONNECTED_LOCAL
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.2507] manager: NetworkManager state is now CONNECTED_SITE
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.2517] policy: set 'Wired connection 1' (enp1s0u1u3i4) as def>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.2533] dns-mgr: Writing DNS information to /sbin/resolvconf
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.3491] device (enp1s0u1u3i4): Activation: successful, device >
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.5335] device (eth0): carrier: link connected
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.5378] device (eth0): state change: unavailable -> disconnect>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.5465] policy: auto-activating connection 'Wired connection 2>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.5645] device (eth0): Activation: starting connection 'Wired >
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.5657] device (eth0): state change: disconnected -> prepare (>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.5679] manager: NetworkManager state is now CONNECTING
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.5692] device (eth0): state change: prepare -> config (reason>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.5717] device (eth0): state change: config -> ip-config (reas>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.5734] dhcp4 (eth0): activation: beginning transaction (timeo>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.6015] dhcp4 (eth0): state changed unknown -> bound, address=>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.6066] device (eth0): state change: ip-config -> ip-check (re>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.6234] device (eth0): state change: ip-check -> secondaries (>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.6242] device (eth0): state change: secondaries -> activated >
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.6270] manager: NetworkManager state is now CONNECTED_SITE
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.6287] dns-mgr: Writing DNS information to /sbin/resolvconf
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.6926] device (eth0): Activation: successful, device activate>
Jul 09 21:02:13 e7d5af5 NetworkManager[1554]: <info>  [1657400533.6952] manager: startup complete
Jul 09 21:02:14 e7d5af5 NetworkManager[1554]: <info>  [1657400534.2825] modem-manager: ModemManager available
Jul 09 21:02:15 e7d5af5 NetworkManager[1554]: <info>  [1657400535.2176] policy: set 'Wired connection 2' (eth0) as default for>
Jul 09 21:02:15 e7d5af5 NetworkManager[1554]: <info>  [1657400535.4807] dhcp6 (enp1s0u1u3i4): activation: beginning transactio>
Jul 09 21:02:15 e7d5af5 NetworkManager[1554]: <info>  [1657400535.4849] policy: set 'Wired connection 1' (enp1s0u1u3i4) as def>
Jul 09 21:02:15 e7d5af5 NetworkManager[1554]: <info>  [1657400535.4864] dns-mgr: Writing DNS information to /sbin/resolvconf
Jul 09 21:02:15 e7d5af5 NetworkManager[1554]: <info>  [1657400535.5541] dhcp6 (enp1s0u1u3i4): state changed unknown -> bound
Jul 09 21:02:15 e7d5af5 NetworkManager[1554]: <info>  [1657400535.7832] device (veth7eb0370): carrier: link connected
Jul 09 21:02:15 e7d5af5 NetworkManager[1554]: <info>  [1657400535.7865] device (br-98d2b6378cb4): carrier: link connected
Jul 09 21:02:28 e7d5af5 NetworkManager[1554]: <info>  [1657400548.8242] policy: set 'Wired connection 2' (eth0) as default for>
Jul 09 21:02:28 e7d5af5 NetworkManager[1554]: <info>  [1657400548.9026] manager: NetworkManager state is now CONNECTED_GLOBAL
Jul 09 21:02:31 e7d5af5 NetworkManager[1554]: <info>  [1657400551.1206] manager: (resin-vpn): new Tun device (/org/freedesktop>
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.0493] device (enp1s0u1u3i4): state change: activated -> unma>
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.0858] dhcp4 (enp1s0u1u3i4): canceled DHCP transaction
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.0860] dhcp4 (enp1s0u1u3i4): state changed bound -> terminated
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.0875] dhcp6 (enp1s0u1u3i4): canceled DHCP transaction
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.0877] dhcp6 (enp1s0u1u3i4): state changed bound -> terminated
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.0917] dns-mgr: Writing DNS information to /sbin/resolvconf
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.1376] dns-mgr: Writing DNS information to /sbin/resolvconf
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.1814] policy: set 'Wired connection 2' (eth0) as default for>
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.1874] modem["ttyUSB2"]: modem state changed, 'disabled' --> >
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.1885] manager: (ttyUSB2): new Broadband device (/org/freedes>
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.1893] device (ttyUSB2): state change: unmanaged -> unavailab>
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.1903] device (ttyUSB2): modem state 'enabling'
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.1927] device (ttyUSB2): state change: unavailable -> disconn>
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.1940] policy: auto-activating connection 'T-Mobile' (>
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.1957] device (ttyUSB2): Activation: starting connection 'T-M>
Jul 09 21:03:00 e7d5af5 NetworkManager[1554]: <info>  [1657400580.1962] device (ttyUSB2): state change: disconnected -> prepar>
Jul 09 21:03:06 e7d5af5 NetworkManager[1554]: <info>  [1657400586.5335] modem["ttyUSB2"]: modem state changed, 'enabling' --> >
Jul 09 21:03:06 e7d5af5 NetworkManager[1554]: <info>  [1657400586.6554] modem["ttyUSB2"]: modem state changed, 'enabled' --> '>
Jul 09 21:03:06 e7d5af5 NetworkManager[1554]: <info>  [1657400586.6894] modem["ttyUSB2"]: modem state changed, 'registered' -->
Jul 09 21:03:06 e7d5af5 NetworkManager[1554]: <info>  [1657400586.7787] modem["ttyUSB2"]: modem state changed, 'connecting' -->
Jul 09 21:03:06 e7d5af5 NetworkManager[1554]: <info>  [1657400586.7872] device (ttyUSB2): state change: prepare -> config (rea>
Jul 09 21:03:06 e7d5af5 NetworkManager[1554]: <info>  [1657400586.7885] device (ttyUSB2): state change: config -> ip-config (r>
Jul 09 21:03:06 e7d5af5 NetworkManager[1554]: <info>  [1657400586.7893] modem["ttyUSB2"]: using modem-specified IP timeout: 20>
Jul 09 21:03:06 e7d5af5 NetworkManager[1554]: <info>  [1657400586.7931] loaded PPP plugin /usr/lib/NetworkManager/1.32.12/libn>
Jul 09 21:03:06 e7d5af5 NetworkManager[1554]: <info>  [1657400586.7945] ppp-manager: starting PPP connection
Jul 09 21:03:06 e7d5af5 NetworkManager[1554]: <info>  [1657400586.8072] ppp-manager: pppd started with pid 2563
Jul 09 21:03:06 e7d5af5 pppd[2563]: Plugin /usr/lib/pppd/2.4.7/nm-pppd-plugin.so loaded.
Jul 09 21:03:06 e7d5af5 NetworkManager[2563]: Plugin /usr/lib/pppd/2.4.7/nm-pppd-plugin.so loaded.
Jul 09 21:03:06 e7d5af5 pppd[2563]: nm-ppp-plugin: initializing
Jul 09 21:03:06 e7d5af5 pppd[2563]: pppd 2.4.9 started by root, uid 0
Jul 09 21:03:06 e7d5af5 pppd[2563]: nm-ppp-plugin: status 3 / phase 'serial connection'
Jul 09 21:03:06 e7d5af5 pppd[2563]: Using interface ppp0
Jul 09 21:03:06 e7d5af5 NetworkManager[2563]: Using interface ppp0
Jul 09 21:03:06 e7d5af5 pppd[2563]: Connect: ppp0 <--> /dev/ttyUSB2
Jul 09 21:03:06 e7d5af5 NetworkManager[2563]: Connect: ppp0 <--> /dev/ttyUSB2
Jul 09 21:03:06 e7d5af5 pppd[2563]: nm-ppp-plugin: status 5 / phase 'establish'
Jul 09 21:03:06 e7d5af5 NetworkManager[1554]: <info>  [1657400586.8761] manager: (ppp0): new Ppp device (/org/freedesktop/Netw>
Jul 09 21:03:07 e7d5af5 pppd[2563]: nm-ppp-plugin: status 6 / phase 'authenticate'
Jul 09 21:03:07 e7d5af5 pppd[2563]: nm-ppp-plugin: passwd-hook, requesting credentials...
Jul 09 21:03:07 e7d5af5 pppd[2563]: nm-ppp-plugin: got credentials from NetworkManager
Jul 09 21:03:07 e7d5af5 pppd[2563]: CHAP authentication succeeded
Jul 09 21:03:07 e7d5af5 pppd[2563]: CHAP authentication succeeded
Jul 09 21:03:07 e7d5af5 pppd[2563]: nm-ppp-plugin: status 8 / phase 'network'
Jul 09 21:03:07 e7d5af5 NetworkManager[2563]: CHAP authentication succeeded
Jul 09 21:03:07 e7d5af5 NetworkManager[2563]: CHAP authentication succeeded
Jul 09 21:03:07 e7d5af5 pppd[2563]: Could not determine remote IP address: defaulting to <redacted>
Jul 09 21:03:07 e7d5af5 NetworkManager[2563]: Could not determine remote IP address: defaulting to <redacted>
Jul 09 21:03:07 e7d5af5 pppd[2563]: Failed to create /etc/ppp/resolv.conf: Read-only file system
Jul 09 21:03:07 e7d5af5 NetworkManager[2563]: Failed to create /etc/ppp/resolv.conf: Read-only file system
Jul 09 21:03:07 e7d5af5 pppd[2563]: local  IP address <redacted>
Jul 09 21:03:07 e7d5af5 NetworkManager[2563]: local  IP address <redacted>
Jul 09 21:03:07 e7d5af5 NetworkManager[2563]: remote IP address <redacted>
Jul 09 21:03:07 e7d5af5 NetworkManager[2563]: primary   DNS address <redacted>
Jul 09 21:03:07 e7d5af5 NetworkManager[2563]: secondary DNS address <redacted>
Jul 09 21:03:07 e7d5af5 pppd[2563]: nm-ppp-plugin: status 9 / phase 'running'
Jul 09 21:03:07 e7d5af5 pppd[2563]: remote IP address <redacted>
Jul 09 21:03:07 e7d5af5 pppd[2563]: primary   DNS address <redacted>
Jul 09 21:03:07 e7d5af5 pppd[2563]: secondary DNS address <redacted>
Jul 09 21:03:07 e7d5af5 NetworkManager[1554]: <info>  [1657400587.9599] device (ppp0): state change: unmanaged -> unavailable >
Jul 09 21:03:07 e7d5af5 pppd[2563]: nm-ppp-plugin: ip-up event
Jul 09 21:03:07 e7d5af5 pppd[2563]: nm-ppp-plugin: sending IPv4 config to NetworkManager...
Jul 09 21:03:07 e7d5af5 NetworkManager[1554]: <info>  [1657400587.9747] device (ppp0): state change: unavailable -> disconnect>
Jul 09 21:03:07 e7d5af5 NetworkManager[1554]: <info>  [1657400587.9779] ppp-manager: (IPv4 Config Get) reply received.
Jul 09 21:03:07 e7d5af5 NetworkManager[1554]: <info>  [1657400587.9855] device (ttyUSB2): state change: ip-config -> ip-check >
Jul 09 21:03:07 e7d5af5 NetworkManager[1554]: <info>  [1657400587.9951] device (ttyUSB2): state change: ip-check -> secondarie>
Jul 09 21:03:07 e7d5af5 NetworkManager[1554]: <info>  [1657400587.9970] device (ttyUSB2): state change: secondaries -> activat>
Jul 09 21:03:08 e7d5af5 NetworkManager[1554]: <info>  [1657400588.0033] dns-mgr: Writing DNS information to /sbin/resolvconf
Jul 09 21:03:08 e7d5af5 NetworkManager[1554]: <info>  [1657400588.0509] device (ttyUSB2): Activation: successful, device activ>
Jul 09 21:03:37 e7d5af5 pppd[2563]: IPV6CP: timeout sending Config-Requests
Jul 09 21:03:37 e7d5af5 NetworkManager[2563]: IPV6CP: timeout sending Config-Requests
Jul 09 21:09:22 e7d5af5 NetworkManager[1554]: <info>  [1657400962.4522] device (wlan0): set-hw-addr: set MAC address to BE:E3:>
Jul 09 21:09:22 e7d5af5 NetworkManager[1554]: <info>  [1657400962.4769] device (wlan0): supplicant interface state: disconnect>
Jul 09 21:09:22 e7d5af5 NetworkManager[1554]: <info>  [1657400962.4773] device (wlan0): supplicant interface state: interface_>
Jul 09 21:16:17 e7d5af5 NetworkManager[1554]: <info>  [1657401377.4604] device (wlan0): set-hw-addr: set MAC address to 7A:3F:>
Jul 09 21:16:17 e7d5af5 NetworkManager[1554]: <info>  [1657401377.4852] device (wlan0): supplicant interface state: inactive ->
Jul 09 21:16:17 e7d5af5 NetworkManager[1554]: <info>  [1657401377.5137] device (wlan0): supplicant interface state: interface_>
root@e7d5af5:~# 

My questions are:

  1. What is the private 192.168.255.0/24 network on balenaOS for? I figured this out; this is how the modem works in ECM mode.
  2. I could use some guidance on next steps to resolve my issue.

Update 1

The above issue is with the modem configured in ECM mode. I noticed that I got an IPv4 address via PPP with T-Mobile. When I switched to QMI mode, T-Mobile will only issue an IPv6 address.

So, with ECM mode, I get an IPv4 address from the carrier via PPP. I can only imagine that this will be slow but, from what I understand, enp1s0u1u3i4 appears to be an inside interface and can be used to access the web interface of the Qualcom modem configuration. With the Sixfab Core SIM (using AT&T Twilio) this network would route through cellular and was the default connection. How can I get to this state in ECM with PPP with T-Mobile?

If I switch to QMI mode, I only get an IPv6 address from the carrier. Does Balena support an IPv6 only connection? Is there a way to NAT the IPv4 traffic out of that IPv6 only connection and make that the primary connection vs eth0?

1 Like

Hello @jmcombs first of all apologizes for the delay answering this message!

What’s the situation of your project right now?

Did you check the metric of every type of connectivity to see which one have priority (eth0 vs cellular)?

@mpous I have this working with a standalone Ubuntu 22.04 Server image with NetworkManager and ModemManager. I just tried my working NM config this morning with Balena and it did not come up and register. I don’t believe Balena gives me console access to find out why it’s not registering. I can say this:

My cellular connection is a T-Mobile US LTE data connection. T-Mobile’s cellular is IPv6 only. They implement CG-NAT in order to allow access to IPv4 networks so it’s important that I use their IPv6 DNS servers and routing.

I am purposely setting the default route to go through cellular and not local LAN. In my use cases, the Balena device will be on a LAN that is isolated so cellular is the connection.

Here is my cellular configuration:

[connection]
id=cellular
type=gsm
interface-name=cdc-wdm0
autoconnect=yes

[ipv4]
method=link-local

[ipv6]
method=auto
addr-gen-mode=eui64

[gsm]
apn="fast.t-mobile.com"

Here is my eth01 configuration:

[connection]
id=internal
type=802-3-ethernet
interface-name=eth0
autoconnect=yes

[ipv4]
method=manual
dns=192.168.7.197
addresses=192.168.7.254/24

[ipv6]
method=shared

All the tests I have been testing on cellular networks, we disable IPv6. Let me ask internally why.

Let’s stay connected!

Hey Jeremy, using a balenaOS development image will provide you with SSH access so you can access logs and debug this on-device. You can download a development image either using the balenaCloud dashboard and creating a fleet, and then selecting the development image, or installing the balena CLI as described in balena-cli/INSTALL.md at master · balena-io/balena-cli · GitHub

You can then download an OS release with:

balena os download --version <version> --output myimage.img

And configure it as development mode with:

balena os configure --dev 

You can list available versions with balena os versions <device-type>, and supported device types with balena devices supported