Hi!
We are currently experiencing issues with one of our devices that is deployed in a production enviroment.
It’s a Balena Fin (CM3), with a SIM7600G-H the balena was working yesterday, on the mobile network, but today it seems that it only wants to go into a VPN Only state.
We had an issue where the computer wanted to connect over ethernet to a plotter so we changed the priorities of the route table. but now the cellular interface has a metric of 20050, when it should have only 50.
Clearly it is somewhat working, as it is connected to the cellular network and APN. but something is certainly wrong as it’s not actually connecting to the internet.
UUID
62d970b
SUPERVISOR VERSION
14.2.16
HOST OS VERSION
balenaOS 2.80.3+rev1
root@62d970b:~# journalctl -u balena-supervisor --no-pager
-- Logs begin at Wed 2023-06-14 16:38:11 UTC, end at Wed 2023-06-14 17:25:23 UTC. --
Jun 14 16:40:49 62d970b balena-supervisor[3098]: [event] Event: Device state report failure {"error":"ESOCKETTIMEDOUT"}
Jun 14 16:40:49 62d970b balena-supervisor[3098]: [info] Retrying current state report in 900 seconds
Jun 14 16:42:12 62d970b balena-supervisor[3098]: [api] GET /v1/healthy 200 - 16.507 ms
Jun 14 16:42:52 62d970b balena-supervisor[3098]: [error] LogBackend: unexpected error: Error: read ETIMEDOUT
Jun 14 16:42:52 62d970b balena-supervisor[3098]: [error] at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20)
Jun 14 16:44:33 62d970b balena-supervisor[3098]: [debug] Attempting container log timestamp flush...
Jun 14 16:44:33 62d970b balena-supervisor[3098]: [debug] Container log timestamp flush complete
Jun 14 16:47:14 62d970b balena-supervisor[3098]: [api] GET /v1/healthy 200 - 17.212 ms
Jun 14 16:49:41 62d970b balena-supervisor[3098]: [api] GET /v1/device 200 - 117.065 ms
Jun 14 16:49:44 62d970b balena-supervisor[3098]: [api] GET /v1/device 200 - 84.839 ms
Jun 14 16:49:50 62d970b balena-supervisor[3098]: [api] GET /v1/device 200 - 78.102 ms
Jun 14 16:49:55 62d970b balena-supervisor[3098]: [api] GET /v1/device 200 - 80.547 ms
Jun 14 16:50:01 62d970b balena-supervisor[3098]: [api] GET /v1/device 200 - 94.351 ms
Jun 14 16:50:06 62d970b balena-supervisor[3098]: [api] GET /v1/device 200 - 86.283 ms
Jun 14 16:50:12 62d970b balena-supervisor[3098]: [api] GET /v1/device 200 - 76.329 ms
Jun 14 16:50:17 62d970b balena-supervisor[3098]: [api] GET /v1/device 200 - 72.577 ms
Jun 14 16:50:23 62d970b balena-supervisor[3098]: [api] GET /v1/device 200 - 85.370 ms
Jun 14 16:52:15 62d970b balena-supervisor[3098]: [api] GET /v1/healthy 200 - 12.350 ms
Jun 14 16:52:54 62d970b balena-supervisor[3098]: [api] GET /v1/healthy 200 - 4.858 ms
Jun 14 16:54:33 62d970b balena-supervisor[3098]: [debug] Attempting container log timestamp flush...
Jun 14 16:54:33 62d970b balena-supervisor[3098]: [debug] Container log timestamp flush complete
Jun 14 16:57:16 62d970b balena-supervisor[3098]: [api] GET /v1/healthy 200 - 12.401 ms
Jun 14 16:57:47 62d970b balena-supervisor[3098]: [event] Event: Device state report failure {"error":"ESOCKETTIMEDOUT"}
Jun 14 16:57:47 62d970b balena-supervisor[3098]: [info] Retrying current state report in 900 seconds
Jun 14 16:59:46 62d970b balena-supervisor[3098]: [error] LogBackend: unexpected error: Error: read ETIMEDOUT
Jun 14 16:59:46 62d970b balena-supervisor[3098]: [error] at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20)
Jun 14 17:02:18 62d970b balena-supervisor[3098]: [api] GET /v1/healthy 200 - 12.356 ms
Jun 14 17:04:33 62d970b balena-supervisor[3098]: [debug] Attempting container log timestamp flush...
Jun 14 17:04:33 62d970b balena-supervisor[3098]: [debug] Container log timestamp flush complete
Jun 14 17:07:19 62d970b balena-supervisor[3098]: [api] GET /v1/healthy 200 - 17.720 ms
Jun 14 17:11:08 62d970b balena-supervisor[3098]: [info] VPN connection is not active.
Jun 14 17:11:18 62d970b balena-supervisor[3098]: [info] Waiting for connectivity...
Jun 14 17:11:22 62d970b balena-supervisor[3098]: [info] VPN connection is active.
Jun 14 17:11:22 62d970b balena-supervisor[3098]: [info] VPN connection is active.
Jun 14 17:11:22 62d970b balena-supervisor[3098]: [info] VPN connection is active.
Jun 14 17:11:22 62d970b balena-supervisor[3098]: [info] VPN connection is active.
Jun 14 17:11:28 62d970b balena-supervisor[3098]: [info] Internet Connectivity: OK
Jun 14 17:12:20 62d970b balena-supervisor[3098]: [api] GET /v1/healthy 200 - 14.113 ms
Jun 14 17:14:33 62d970b balena-supervisor[3098]: [debug] Attempting container log timestamp flush...
Jun 14 17:14:33 62d970b balena-supervisor[3098]: [debug] Container log timestamp flush complete
Jun 14 17:14:46 62d970b balena-supervisor[3098]: [event] Event: Device state report failure {"error":"ESOCKETTIMEDOUT"}
Jun 14 17:14:46 62d970b balena-supervisor[3098]: [info] Retrying current state report in 900 seconds
Jun 14 17:16:45 62d970b balena-supervisor[3098]: [error] LogBackend: unexpected error: Error: read ETIMEDOUT
Jun 14 17:16:45 62d970b balena-supervisor[3098]: [error] at TLSWrap.onStreamRead (node:internal/stream_base_commons:217:20)
Jun 14 17:17:22 62d970b balena-supervisor[3098]: [api] GET /v1/healthy 200 - 11.083 ms
Jun 14 17:22:23 62d970b balena-supervisor[3098]: [api] GET /v1/healthy 200 - 9.976 ms
Jun 14 17:24:33 62d970b balena-supervisor[3098]: [debug] Attempting container log timestamp flush...
Jun 14 17:24:33 62d970b balena-supervisor[3098]: [debug] Container log timestamp flush complete
root@62d970b:~# mmcli -m 0
-----------------------------------
General | dbus path: /org/freedesktop/ModemManager1/Modem/0
| device id: b7c478ca3fa2693316fa889093c46f317417e398
-----------------------------------
Hardware | manufacturer: QUALCOMM INCORPORATED
| model: SIMCOM_SIM7600G-H
| firmware revision: LE20B03SIM7600M22
| carrier config: VF_Italy_VoLTE
| carrier config revision: 0501043A
| h/w revision: 10000
| supported: gsm-umts, lte
| current: gsm-umts, lte
| equipment id: 868822042018059
-----------------------------------
System | device: /sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.2
| drivers: option1, qmi_wwan
| plugin: simtech
| primary port: cdc-wdm0
| ports: cdc-wdm0 (qmi), ttyUSB0 (qcdm), ttyUSB2 (at), wwan0 (net),
| ttyUSB4 (audio), ttyUSB1 (gps), ttyUSB3 (at)
-----------------------------------
Numbers | own: 393452223818
-----------------------------------
Status | lock: sim-pin2
| unlock retries: sim-pin (3), sim-puk (10), sim-pin2 (3), sim-puk2 (10)
| state: connected
| power state: on
| access tech: lte
| signal quality: 73% (recent)
-----------------------------------
Modes | supported: allowed: 2g; preferred: none
| allowed: 3g; preferred: none
| allowed: 4g; preferred: none
| allowed: 2g, 3g; preferred: 3g
| allowed: 2g, 3g; preferred: 2g
| allowed: 2g, 4g; preferred: 4g
| allowed: 2g, 4g; preferred: 2g
| allowed: 3g, 4g; preferred: 4g
| allowed: 3g, 4g; preferred: 3g
| allowed: 2g, 3g, 4g; preferred: 4g
| allowed: 2g, 3g, 4g; preferred: 3g
| allowed: 2g, 3g, 4g; preferred: 2g
| current: allowed: 2g, 3g, 4g; preferred: 4g
-----------------------------------
Bands | supported: egsm, dcs, pcs, g850, utran-1, utran-4, utran-6, utran-5,
| utran-8, utran-2, eutran-1, eutran-2, eutran-3, eutran-4, eutran-5,
| eutran-7, eutran-8, eutran-12, eutran-13, eutran-18, eutran-19,
| eutran-20, eutran-25, eutran-26, eutran-28, eutran-34, eutran-38,
| eutran-39, eutran-40, eutran-41, eutran-66, utran-19
| current: egsm, dcs, pcs, g850, utran-1, utran-4, utran-6, utran-5,
| utran-8, utran-2, eutran-1, eutran-2, eutran-3, eutran-4, eutran-5,
| eutran-7, eutran-8, eutran-12, eutran-13, eutran-18, eutran-19,
| eutran-20, eutran-25, eutran-26, eutran-28, eutran-34, eutran-38,
| eutran-39, eutran-40, eutran-41, eutran-66, utran-19
-----------------------------------
IP | supported: ipv4, ipv6, ipv4v6
-----------------------------------
3GPP | imei: 868822042018059
| operator id: 22210
| operator name: vodafone IT
| registration: home
-----------------------------------
3GPP EPS | ue mode of operation: csps-1
-----------------------------------
SIM | dbus path: /org/freedesktop/ModemManager1/SIM/0
-----------------------------------
Bearer | dbus path: /org/freedesktop/ModemManager1/Bearer/0
root@62d970b:~# mmcli -b 0
------------------------------------
General | dbus path: /org/freedesktop/ModemManager1/Bearer/0
| type: default
------------------------------------
Status | connected: yes
| suspended: no
| interface: wwan0
| ip timeout: 20
------------------------------------
Properties | apn: mobile.vodafone.it
| roaming: allowed
| ip type: ipv4
------------------------------------
IPv4 configuration | method: static
| address: 100.105.129.9
| prefix: 30
| gateway: 100.105.129.10
| dns: 10.133.47.214, 10.128.28.17
| mtu: 1430
------------------------------------
Statistics | duration: 23640
| bytes rx: 2632144
| bytes tx: 13846733
| attempts: 330
| attempts: 328
| total-duration: 37041
| total-bytes rx: 3433404
| total-bytes tx: 18119672
root@62d970b:~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default _gateway 0.0.0.0 UG 20050 0 0 wwan0
10.42.0.0 * 255.255.255.0 U 600 0 0 wlan0
10.114.101.0 * 255.255.255.0 U 0 0 0 balena0
10.114.102.0 * 255.255.255.0 U 0 0 0 resin-dns
10.114.104.0 * 255.255.255.128 U 0 0 0 supervisor0
52.4.252.97 * 255.255.255.255 UH 0 0 0 resin-vpn
100.105.129.8 * 255.255.255.252 U 50 0 0 wwan0
172.18.0.0 * 255.255.0.0 U 0 0 0 br-c6b4a1e15c8f
172.19.0.0 * 255.255.0.0 U 0 0 0 br-d483a9dfbcdc
root@62d970b:~# nmcli -p -f general device show cdc-wdm0
===============================================================================
Device details (cdc-wdm0)
===============================================================================
GENERAL.DEVICE: cdc-wdm0
GENERAL.TYPE: gsm
GENERAL.NM-TYPE: NMDeviceModem
GENERAL.DBUS-PATH: /org/freedesktop/NetworkManager/Devices/8
GENERAL.VENDOR: --
GENERAL.PRODUCT: --
GENERAL.DRIVER: option1, qmi_wwan
GENERAL.DRIVER-VERSION: --
GENERAL.FIRMWARE-VERSION: --
GENERAL.HWADDR: (unknown)
GENERAL.MTU: 1430
GENERAL.STATE: 100 (connected)
GENERAL.REASON: 0 (No reason given)
GENERAL.IP4-CONNECTIVITY: 3 (limited)
GENERAL.IP6-CONNECTIVITY: 1 (none)
GENERAL.UDI: /org/freedesktop/ModemManager1/Modem/0
GENERAL.PATH: --
GENERAL.IP-IFACE: wwan0
GENERAL.IS-SOFTWARE: no
GENERAL.NM-MANAGED: yes
GENERAL.AUTOCONNECT: yes
GENERAL.FIRMWARE-MISSING: no
GENERAL.NM-PLUGIN-MISSING: no
GENERAL.PHYS-PORT-ID: --
GENERAL.CONNECTION: cellular
GENERAL.CON-UUID: d811a35a-70fc-3716-84e1-a3578d10cc1e
GENERAL.CON-PATH: /org/freedesktop/NetworkManager/ActiveConnection/337
GENERAL.METERED: yes (guessed)
-------------------------------------------------------------------------------
root@62d970b:~# cat /mnt/boot/system-connections/cellular
[connection]
id=cellular
type=gsm
autoconnect=true
[gsm]
pin=1234
apn=mobile.vodafone.it
number=*99#
[serial]
baud=115200
[ipv4]
method=auto
route-metric=50
[ipv6]
addr-gen-mode=stable-privacy
method=disabled