Critical issue with ModemManager

Hi I’ve started encountering an issue where the modem is unable to register itself to the cellular network. This is a major problem for devices running in production.

I’m using the IOT-GATE-IMX8 paired with a SIMCOM_SIM7600G modem.
In this instance the device is running balenaOS 4.1.3+rev2

I’ve written another post about this issue in the past Modem manager doesn't reconnect properly

The issue appears as follows:

  • Operator issues a disconnect to the modem
  • Modem is unable to reconnect, continuously getting a GsmRegistrationTimeout
  • Modem is rendered unusable until reboot

Resetting the modem works in some instances, but sometimes it has no effect.

Sometimes, enabling and disabling the modem mitigates the issue.

But ultimately in many cases enabling/disabling the modem does not work!

Below issue occurs both when enabling and disabling the modem and or attempting to changing the power state of the modem.

root@b94de6a:~# mmcli -m 0 -e
error: couldn't enable the modem: 'GDBus.Error:org.freedesktop.ModemManager1.Error.Core.Retry: Invalid transition'

Here’s some logs after restarting the ModemManager service (which does not mitigate the issue)

root@b94de6a:~# systemctl stop ModemManager.service
root@b94de6a:~# systemctl start ModemManager.service
root@b94de6a:~# journalctl | grep Modem
Feb 28 12:16:17 b94de6a systemd[1]: ModemManager.service: Killing process 1646 (gmain) with signal SIGKILL.
Feb 28 12:16:18 b94de6a dbus-daemon[1208]: [system] Activating via systemd: service name='org.freedesktop.ModemManager1' unit='dbus-org.freedesktop.ModemManager1.service' requested by ':1.726' (uid=0 pid=616999 comm="./NetworkManager")
Feb 28 12:16:18 b94de6a ModemManager[1002189]: ModemManager (version 1.20.6) starting in system bus...
Feb 28 12:16:18 b94de6a dbus-daemon[1208]: [system] Successfully activated service 'org.freedesktop.ModemManager1'
Feb 28 12:16:21 b94de6a ModemManager[1002189]: [device /sys/devices/platform/soc@0/32c00000.bus/32e50000.usb/ci_hdrc.1/usb2/2-1/2-1.5] creating modem with plugin 'simtech' and '7' ports
Feb 28 12:16:21 b94de6a ModemManager[1002189]: [base-manager] modem for device '/sys/devices/platform/soc@0/32c00000.bus/32e50000.usb/ci_hdrc.1/usb2/2-1/2-1.5' successfully created
Feb 28 12:16:21 b94de6a ModemManager[1002189]: [base-manager] couldn't check support for device '/sys/devices/platform/soc@0/30800000.bus/30be0000.ethernet': not supported by any plugin
Feb 28 12:16:21 b94de6a ModemManager[1002189]: [base-manager] couldn't check support for device '/sys/devices/platform/soc@0/32c00000.bus/32e50000.usb/ci_hdrc.1/usb2/2-1/2-1.1': not supported by any plugin
Feb 28 12:16:21 b94de6a ModemManager[1002189]: [base-manager] couldn't check support for device '/sys/devices/platform/soc@0/33800000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0': not supported by any plugin
Feb 28 12:16:22 b94de6a ModemManager[1002189]: [modem0/sim0] couldn't load operator identifier: Couldn't get home network: QMI protocol error (16): 'NotProvisioned'
Feb 28 12:16:22 b94de6a ModemManager[1002189]: [modem0/sim0] couldn't load operator name: Couldn't get home network: QMI protocol error (16): 'NotProvisioned'
Feb 28 12:16:22 b94de6a ModemManager[1002189]: [modem0/sim0] couldn't load GID1: Couldn't read data from UIM: QMI protocol error (80): 'SimFileNotFound'
Feb 28 12:16:22 b94de6a ModemManager[1002189]: [modem0/sim0] couldn't load GID2: Couldn't read data from UIM: QMI protocol error (80): 'SimFileNotFound'
Feb 28 12:16:22 b94de6a ModemManager[1002189]: [modem0] couldn't load list of own numbers: Couldn't get MSISDN: QMI protocol error (16): 'NotProvisioned'
Feb 28 12:16:22 b94de6a ModemManager[1002189]: [modem0] couldn't load facility locks: QMI message Get Configuration failed: QMI protocol error (94): 'NotSupported'
Feb 28 12:16:23 b94de6a ModemManager[1002189]: [modem0] couldn't load supported assistance data types: LOC service: general failure
Feb 28 12:16:23 b94de6a ModemManager[1002189]: [modem0] state changed (unknown -> disabled)
Feb 28 12:16:28 b94de6a ModemManager[1002189]: caught signal, shutting down...
Feb 28 12:16:28 b94de6a ModemManager[1002189]: Cannot read from istream: connection broken
Feb 28 12:16:28 b94de6a ModemManager[1002189]: [modem0] port 'cdc-wdm0' no longer controllable, reprobing
Feb 28 12:16:28 b94de6a ModemManager[1002189]: ModemManager is shut down
Feb 28 12:16:29 b94de6a NetworkManager[1545]: <info>  [1709122589.9008] modem-manager: ModemManager now available
Feb 28 12:16:29 b94de6a NetworkManager[1545]: <info>  [1709122589.9063] modem-manager: ModemManager no longer available
Feb 28 12:16:42 b94de6a ModemManager[1002241]: ModemManager (version 1.20.6) starting in system bus...
Feb 28 12:16:44 b94de6a ModemManager[1002241]: [device /sys/devices/platform/soc@0/32c00000.bus/32e50000.usb/ci_hdrc.1/usb2/2-1/2-1.5] creating modem with plugin 'simtech' and '7' ports
Feb 28 12:16:44 b94de6a ModemManager[1002241]: [base-manager] modem for device '/sys/devices/platform/soc@0/32c00000.bus/32e50000.usb/ci_hdrc.1/usb2/2-1/2-1.5' successfully created
Feb 28 12:16:45 b94de6a ModemManager[1002241]: [base-manager] couldn't check support for device '/sys/devices/platform/soc@0/30800000.bus/30be0000.ethernet': not supported by any plugin
Feb 28 12:16:45 b94de6a ModemManager[1002241]: [base-manager] couldn't check support for device '/sys/devices/platform/soc@0/32c00000.bus/32e50000.usb/ci_hdrc.1/usb2/2-1/2-1.1': not supported by any plugin
Feb 28 12:16:45 b94de6a ModemManager[1002241]: [base-manager] couldn't check support for device '/sys/devices/platform/soc@0/33800000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0': not supported by any plugin
Feb 28 12:16:46 b94de6a ModemManager[1002241]: [modem0/sim0] couldn't load operator identifier: Couldn't get home network: QMI protocol error (16): 'NotProvisioned'
Feb 28 12:16:46 b94de6a ModemManager[1002241]: [modem0/sim0] couldn't load operator name: Couldn't get home network: QMI protocol error (16): 'NotProvisioned'
Feb 28 12:16:46 b94de6a ModemManager[1002241]: [modem0/sim0] couldn't load GID1: Couldn't read data from UIM: QMI protocol error (80): 'SimFileNotFound'
Feb 28 12:16:46 b94de6a ModemManager[1002241]: [modem0/sim0] couldn't load GID2: Couldn't read data from UIM: QMI protocol error (80): 'SimFileNotFound'
Feb 28 12:16:46 b94de6a ModemManager[1002241]: [modem0] couldn't load list of own numbers: Couldn't get MSISDN: QMI protocol error (16): 'NotProvisioned'
Feb 28 12:16:46 b94de6a ModemManager[1002241]: [modem0] couldn't load facility locks: QMI message Get Configuration failed: QMI protocol error (94): 'NotSupported'
Feb 28 12:16:46 b94de6a ModemManager[1002241]: [modem0] couldn't load supported assistance data types: LOC service: general failure
Feb 28 12:16:46 b94de6a ModemManager[1002241]: [modem0] state changed (unknown -> disabled)
Feb 28 12:16:48 b94de6a NetworkManager[1545]: <warn>  [1709122608.1847] modem-broadband[cdc-wdm0]: failed to enable modem: GDBus.Error:org.freedesktop.DBus.Error.UnknownMethod: Method "Enable" with signature "b" on interface "org.freedesktop.ModemManager1.Modem" doesn't exist
Feb 28 12:16:48 b94de6a NetworkManager[1545]: <info>  [1709122608.2123] modem-manager: ModemManager now available
Feb 28 12:16:48 b94de6a NetworkManager[1545]: <warn>  [1709122608.2125] modem-manager: modem with path /org/freedesktop/ModemManager1/Modem/0 already exists, ignoring
Feb 28 12:16:54 b94de6a ModemManager[1002241]: [base-manager] couldn't check support for device '/sys/devices/platform/soc@0/30800000.bus/30be0000.ethernet': not supported by any plugin
Feb 28 12:16:54 b94de6a ModemManager[1002241]: [base-manager] couldn't check support for device '/sys/devices/platform/soc@0/32c00000.bus/32e50000.usb/ci_hdrc.1/usb2/2-1/2-1.1': not supported by any plugin
Feb 28 12:16:54 b94de6a ModemManager[1002241]: [base-manager] couldn't check support for device '/sys/devices/platform/soc@0/33800000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0': not supported by any plugin
Feb 28 12:16:56 b94de6a ModemManager[1002241]: [modem0] state changed (disabled -> enabling)
Feb 28 12:16:56 b94de6a ModemManager[1002241]: [modem0] couldn't enable interface: 'Invalid transition'
Feb 28 12:17:00 b94de6a ModemManager[1002241]: [modem0] 3GPP registration state changed (unknown -> unknown)
Feb 28 12:17:00 b94de6a ModemManager[1002241]: [modem0] state changed (enabling -> disabled)
Feb 28 12:24:24 b94de6a ModemManager[1002241]: [modem0] state changed (disabled -> enabling)
Feb 28 12:24:24 b94de6a ModemManager[1002241]: [modem0] couldn't enable interface: 'Invalid transition'
Feb 28 12:24:24 b94de6a ModemManager[1002241]: [modem0] state changed (enabling -> disabled)
Feb 28 12:27:29 b94de6a ModemManager[1002241]: [modem0] state changed (disabled -> enabling)
Feb 28 12:27:29 b94de6a ModemManager[1002241]: [modem0] couldn't enable interface: 'Invalid transition'
Feb 28 12:27:29 b94de6a ModemManager[1002241]: [modem0] state changed (enabling -> disabled)
Feb 28 12:28:10 b94de6a ModemManager[1002241]: caught signal, shutting down...
Feb 28 12:28:10 b94de6a NetworkManager[1545]: <info>  [1709123290.4914] modem-manager: ModemManager no longer available
Feb 28 12:28:10 b94de6a ModemManager[1002241]: ModemManager is shut down
Feb 28 12:28:23 b94de6a ModemManager[1003037]: ModemManager (version 1.20.6) starting in system bus...
Feb 28 12:28:23 b94de6a NetworkManager[1545]: <info>  [1709123303.7315] modem-manager: ModemManager now available
Feb 28 12:28:26 b94de6a ModemManager[1003037]: [device /sys/devices/platform/soc@0/32c00000.bus/32e50000.usb/ci_hdrc.1/usb2/2-1/2-1.5] creating modem with plugin 'simtech' and '7' ports
Feb 28 12:28:26 b94de6a ModemManager[1003037]: [base-manager] modem for device '/sys/devices/platform/soc@0/32c00000.bus/32e50000.usb/ci_hdrc.1/usb2/2-1/2-1.5' successfully created
Feb 28 12:28:26 b94de6a ModemManager[1003037]: [base-manager] couldn't check support for device '/sys/devices/platform/soc@0/30800000.bus/30be0000.ethernet': not supported by any plugin
Feb 28 12:28:26 b94de6a ModemManager[1003037]: [base-manager] couldn't check support for device '/sys/devices/platform/soc@0/32c00000.bus/32e50000.usb/ci_hdrc.1/usb2/2-1/2-1.1': not supported by any plugin
Feb 28 12:28:26 b94de6a ModemManager[1003037]: [base-manager] couldn't check support for device '/sys/devices/platform/soc@0/33800000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0': not supported by any plugin
Feb 28 12:28:27 b94de6a ModemManager[1003037]: [modem0/sim0] couldn't load operator identifier: Couldn't get home network: QMI protocol error (16): 'NotProvisioned'
Feb 28 12:28:27 b94de6a ModemManager[1003037]: [modem0/sim0] couldn't load operator name: Couldn't get home network: QMI protocol error (16): 'NotProvisioned'
Feb 28 12:28:27 b94de6a ModemManager[1003037]: [modem0/sim0] couldn't load GID1: Couldn't read data from UIM: QMI protocol error (80): 'SimFileNotFound'
Feb 28 12:28:27 b94de6a ModemManager[1003037]: [modem0/sim0] couldn't load GID2: Couldn't read data from UIM: QMI protocol error (80): 'SimFileNotFound'
Feb 28 12:28:27 b94de6a ModemManager[1003037]: [modem0] couldn't load list of own numbers: Couldn't get MSISDN: QMI protocol error (16): 'NotProvisioned'
Feb 28 12:28:27 b94de6a ModemManager[1003037]: [modem0] couldn't load facility locks: QMI message Get Configuration failed: QMI protocol error (94): 'NotSupported'
Feb 28 12:28:27 b94de6a ModemManager[1003037]: [modem0] couldn't load supported assistance data types: LOC service: general failure
Feb 28 12:28:27 b94de6a ModemManager[1003037]: [modem0] state changed (unknown -> disabled)
Feb 28 12:28:27 b94de6a ModemManager[1003037]: [modem0] state changed (disabled -> enabling)
Feb 28 12:28:28 b94de6a ModemManager[1003037]: [modem0] couldn't enable interface: 'Invalid transition'
Feb 28 12:28:28 b94de6a ModemManager[1003037]: [modem0] 3GPP registration state changed (unknown -> unknown)
Feb 28 12:28:28 b94de6a ModemManager[1003037]: [modem0] state changed (enabling -> disabled)
Feb 28 12:28:32 b94de6a NetworkManager[1545]: <warn>  [1709123312.5451] modem-broadband[cdc-wdm0]: failed to enable modem: GDBus.Error:org.freedesktop.ModemManager1.Error.Core.Retry: Invalid transition

Yeah… We’ve also been seeing critical issues with ModemManager for the past 6mos with the version shipped with balenaOS, we have spent countless hours trying to debug the issue, and keep coming to the same conclusion that something is wrong with the balenaOS intergration…

What makes it so much more frustrating is, that we are unable to upgrade or downgrade the MM package…