mmcli stuck in registered state?

Having a little trouble with mmcli and a sixfab 4g iot hat. Here’s what I’ve got:

Everything seems fine, strong signal, correct network (tmobile), gps fix with accurate lat/long, and even the correct mobile phone number (see Numbers below). But for some reason it remains in Status: state: registered. Which yields a disconnected state in nmcli and no cellular IP address in balenacloud ui.

Any ideas? I had thought perhaps it was an APN problem but I tripled checked that, or that it was a registration problem with iotDataworks, but that also seems fine per the Number matching automatically what they said belonged to my sim.

root@fff19d1:~# mmcli -m 0
  -----------------------------------
  General  |               dbus path: /org/freedesktop/ModemManager1/Modem/0
           |               device id: [redacted]
  -----------------------------------
  Hardware |            manufacturer: QUALCOMM INCORPORATED
           |                   model: QUECTEL Mobile Broadband Module
           |       firmware revision: EG25GGBR07A08M2G
           |          carrier config: Commercial-TMO_VoLTE
           | carrier config revision: 05010505
           |            h/w revision: 10000
           |               supported: gsm-umts, lte
           |                 current: gsm-umts, lte
           |            equipment id: [redacted]
  -----------------------------------
  System   |                  device: /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.3
           |                 drivers: option1, qmi_wwan
           |                  plugin: quectel
           |            primary port: cdc-wdm0
           |                   ports: cdc-wdm0 (qmi), ttyUSB0 (qcdm), ttyUSB2 (at), 
           |                          ttyUSB1 (gps), wwp1s0u1u3i4 (net), ttyUSB3 (at)
  -----------------------------------
  Numbers  |                     own: 1523[redacted]
  -----------------------------------
  Status   |                    lock: sim-pin2
           |          unlock retries: sim-pin (3), sim-puk (10), sim-pin2 (10), sim-puk2 (10)
           |                   state: registered
           |             power state: on
           |             access tech: lte
           |          signal quality: 68% (recent)
  -----------------------------------
  Modes    |               supported: allowed: 2g; preferred: none
           |                          allowed: 3g; preferred: none
           |                          allowed: 4g; preferred: none
           |                          allowed: 2g, 3g; preferred: 3g
           |                          allowed: 2g, 3g; preferred: 2g
           |                          allowed: 2g, 4g; preferred: 4g
           |                          allowed: 2g, 4g; preferred: 2g
           |                          allowed: 3g, 4g; preferred: 4g
           |                          allowed: 3g, 4g; preferred: 3g
           |                          allowed: 2g, 3g, 4g; preferred: 4g
           |                          allowed: 2g, 3g, 4g; preferred: 3g
           |                          allowed: 2g, 3g, 4g; preferred: 2g
           |                 current: allowed: 2g, 3g, 4g; preferred: 4g
  -----------------------------------
  Bands    |               supported: egsm, dcs, pcs, g850, utran-1, utran-4, utran-6, utran-5, 
           |                          utran-8, utran-2, eutran-1, eutran-2, eutran-3, eutran-4, eutran-5, 
           |                          eutran-7, eutran-8, eutran-12, eutran-13, eutran-18, eutran-19, 
           |                          eutran-20, eutran-25, eutran-26, eutran-28, eutran-38, eutran-39, 
           |                          eutran-40, eutran-41, utran-19
           |                 current: egsm, dcs, pcs, g850, utran-1, utran-4, utran-6, utran-5, 
           |                          utran-8, utran-2, eutran-1, eutran-2, eutran-3, eutran-4, eutran-5, 
           |                          eutran-7, eutran-8, eutran-12, eutran-13, eutran-18, eutran-19, 
           |                          eutran-20, eutran-25, eutran-26, eutran-28, eutran-38, eutran-39, 
           |                          eutran-40, eutran-41, utran-19
  -----------------------------------
  IP       |               supported: ipv4, ipv6, ipv4v6
  -----------------------------------
  3GPP     |                    imei: [redacted]
           |             operator id: 310260
           |           operator name: T-Mobile
           |            registration: home
  -----------------------------------
  3GPP EPS |    ue mode of operation: csps-1
  -----------------------------------
  SIM      |               dbus path: /org/freedesktop/ModemManager1/SIM/0
  -----------------------------------
  Bearer   |               dbus path: /org/freedesktop/ModemManager1/Bearer/0

And, nmcli shows no connection on cellular.

> root@fff19d1:~# nmcli
>     cdc-wdm0: disconnected
>             "cdc-wdm0"
>             gsm (option1, qmi_wwan), hw

Update, I should have tried this earlier but I just tested with a different sim from twilio instead and it works. Looks like I need to check in with iotdataworks.
Whoops

I may have spoken a little too soon. Oddly, mmcli shows that I am connected, but it isn’t playing nice with nmcli. Apologies in advance, below output is big. But the main issue I believe is this:

Everything looks fine on the ModemManager side…

mmcli -m 0
state: connected
primary port: cdc-wdm0.

Seems like we’re all good, which is where I left the last post … But, NetworkManager doesn’t seem to agree with that.

nmcli
cdc-wdm0: disconnected

And if I did a little further, it appears to not be able to find the hardware?

nmcli device show
GENERAL.HWADDR: (unknown) GENERAL.STATE: 30 (disconnected).

What could cause the disagreement between ModemManager and NetworkManager?

Full output:

root@fff19d1:~# mmcli -m 0

  -----------------------------------
  General  |               dbus path: /org/freedesktop/ModemManager1/Modem/0
           |               device id: [redacted]
  -----------------------------------
  Hardware |            manufacturer: QUALCOMM INCORPORATED
           |                   model: QUECTEL Mobile Broadband Module
           |       firmware revision: EG25GGBR07A08M2G
           |          carrier config: Commercial-TMO_VoLTE
           | carrier config revision: 05010505
           |            h/w revision: 10000
           |               supported: gsm-umts, lte
           |                 current: gsm-umts, lte
           |            equipment id: [redacted]
  -----------------------------------
  System   |                  device: /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.3
           |                 drivers: option1, qmi_wwan
           |                  plugin: quectel
           |            primary port: cdc-wdm0
           |                   ports: cdc-wdm0 (qmi), ttyUSB0 (qcdm), ttyUSB2 (at), 
           |                          ttyUSB1 (gps), wwp1s0u1u3i4 (net), ttyUSB3 (at)
  -----------------------------------
  Numbers  |                     own: [redacted]
  -----------------------------------
  Status   |                    lock: sim-pin2
           |          unlock retries: sim-pin (3), sim-puk (10), sim-pin2 (10), sim-puk2 (10)
           |                   state: connected
           |             power state: on
           |             access tech: lte
           |          signal quality: 84% (recent)
  -----------------------------------
  Modes    |               supported: allowed: 2g; preferred: none
           |                          allowed: 3g; preferred: none
           |                          allowed: 4g; preferred: none
           |                          allowed: 2g, 3g; preferred: 3g
           |                          allowed: 2g, 3g; preferred: 2g
           |                          allowed: 2g, 4g; preferred: 4g
           |                          allowed: 2g, 4g; preferred: 2g
           |                          allowed: 3g, 4g; preferred: 4g
           |                          allowed: 3g, 4g; preferred: 3g
           |                          allowed: 2g, 3g, 4g; preferred: 4g
           |                          allowed: 2g, 3g, 4g; preferred: 3g
           |                          allowed: 2g, 3g, 4g; preferred: 2g
           |                 current: allowed: 2g, 3g, 4g; preferred: 4g
  -----------------------------------
  Bands    |               supported: egsm, dcs, pcs, g850, utran-1, utran-4, utran-6, utran-5, 
           |                          utran-8, utran-2, eutran-1, eutran-2, eutran-3, eutran-4, eutran-5, 
           |                          eutran-7, eutran-8, eutran-12, eutran-13, eutran-18, eutran-19, 
           |                          eutran-20, eutran-25, eutran-26, eutran-28, eutran-38, eutran-39, 
           |                          eutran-40, eutran-41, utran-19
           |                 current: egsm, dcs, pcs, g850, utran-1, utran-4, utran-6, utran-5, 
           |                          utran-8, utran-2, eutran-1, eutran-2, eutran-3, eutran-4, eutran-5, 
           |                          eutran-7, eutran-8, eutran-12, eutran-13, eutran-18, eutran-19, 
           |                          eutran-20, eutran-25, eutran-26, eutran-28, eutran-38, eutran-39, 
           |                          eutran-40, eutran-41, utran-19
  -----------------------------------
  IP       |               supported: ipv4, ipv6, ipv4v6
  -----------------------------------
  3GPP     |                    imei: [redacted]
           |             operator id: 310260
           |           operator name: Twilio
           |            registration: home
  -----------------------------------
  3GPP EPS |    ue mode of operation: csps-1
  -----------------------------------
  SIM      |               dbus path: /org/freedesktop/ModemManager1/SIM/0
  -----------------------------------
  Bearer   |               dbus path: /org/freedesktop/ModemManager1/Bearer/0
root@fff19d1:~# nmcli 
eth0: connected to Wired connection 1
        "eth0"
        ethernet (bcmgenet), DC:[redacted]:52, hw, mtu 1500
        ip4 default
        inet4 192.168.1.55/24
        route4 0.0.0.0/0
        route4 192.168.1.0/24
        inet6 fd3a:414f:9b6b:1:fef2:34bd:7ac5:6896/64
        inet6 fe80::aec9:dc68:482f:d5d2/64
        route6 fd3a:414f:9b6b::/64
        route6 fd3a:414f:9b6b:1::/64
        route6 ff00::/8
        route6 fe80::/64

wlan0: connected to resin-wifi-01
        "wlan0"
        wifi (brcmfmac), DC:[redacted]:54, hw, mtu 1500
        inet4 192.168.1.69/24
        route4 0.0.0.0/0
        route4 192.168.1.0/24
        inet6 fd3a:414f:9b6b:1:5b2:2a5:c524:2c07/64
        inet6 fe80::741e:6dc3:90bb:c8df/64
        route6 fd3a:414f:9b6b::/64
        route6 fd3a:414f:9b6b:1::/64
        route6 ff00::/8
        route6 fe80::/64

supervisor0: connected (externally) to supervisor0
        "supervisor0"
        bridge, 02:42:D6:BD:E5:74, sw, mtu 1500
        inet4 10.114.104.1/25
        route4 10.114.104.0/25

cdc-wdm0: disconnected
        "cdc-wdm0"
        gsm (option1, qmi_wwan), hw

balena0: unmanaged
        "balena0"
        bridge, 02:42:F1:A7:53:36, sw, mtu 1500

br-859fdb826d9e: unmanaged
        "br-859fdb826d9e"
        bridge, 02:42:5E:B4:3A:53, sw, mtu 1500

resin-dns: unmanaged
        "resin-dns"
        bridge, DE:8B:69:85:D8:38, sw, mtu 1500

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

resin-vpn: unmanaged
        "resin-vpn"
        tun, sw, mtu 1500

DNS configuration:
        servers: 192.168.1.1
        interface: eth0

        servers: 192.168.1.1
        interface: wlan0

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.


root@fff19d1:~# nmcli device show

GENERAL.DEVICE:                         eth0
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         DC:[redacted]:52
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     Wired connection 1
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/3
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         192.168.1.55/24
IP4.GATEWAY:                            192.168.1.1
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.1.1, mt = 100
IP4.ROUTE[2]:                           dst = 192.168.1.0/24, nh = 0.0.0.0, mt = 100
IP4.DNS[1]:                             192.168.1.1
IP6.ADDRESS[1]:                         fd3a:414f:9b6b:1:fef2:34bd:7ac5:6896/64
IP6.ADDRESS[2]:                         fe80::aec9:dc68:482f:d5d2/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fd3a:414f:9b6b::/64, nh = fe80::c78:2dc9:65e6:8f81, mt = 100
IP6.ROUTE[2]:                           dst = fd3a:414f:9b6b:1::/64, nh = ::, mt = 100
IP6.ROUTE[3]:                           dst = ff00::/8, nh = ::, mt = 256, table=255
IP6.ROUTE[4]:                           dst = fe80::/64, nh = ::, mt = 100

GENERAL.DEVICE:                         wlan0
GENERAL.TYPE:                           wifi
GENERAL.HWADDR:                         DC:[redacted]:54
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     resin-wifi-01
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/2
IP4.ADDRESS[1]:                         192.168.1.69/24
IP4.GATEWAY:                            192.168.1.1
IP4.ROUTE[1]:                           dst = 0.0.0.0/0, nh = 192.168.1.1, mt = 600
IP4.ROUTE[2]:                           dst = 192.168.1.0/24, nh = 0.0.0.0, mt = 600
IP4.DNS[1]:                             192.168.1.1
IP6.ADDRESS[1]:                         fd3a:414f:9b6b:1:5b2:2a5:c524:2c07/64
IP6.ADDRESS[2]:                         fe80::741e:6dc3:90bb:c8df/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fd3a:414f:9b6b::/64, nh = fe80::c78:2dc9:65e6:8f81, mt = 600
IP6.ROUTE[2]:                           dst = fd3a:414f:9b6b:1::/64, nh = ::, mt = 600
IP6.ROUTE[3]:                           dst = ff00::/8, nh = ::, mt = 256, table=255
IP6.ROUTE[4]:                           dst = fe80::/64, nh = ::, mt = 600

GENERAL.DEVICE:                         supervisor0
GENERAL.TYPE:                           bridge
GENERAL.HWADDR:                         02:42:D6:BD:E5:74
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected (externally))
GENERAL.CONNECTION:                     supervisor0
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/1
IP4.ADDRESS[1]:                         10.114.104.1/25
IP4.GATEWAY:                            --
IP4.ROUTE[1]:                           dst = 10.114.104.0/25, nh = 0.0.0.0, mt = 0
IP6.GATEWAY:                            --

GENERAL.DEVICE:                         cdc-wdm0
GENERAL.TYPE:                           gsm
GENERAL.HWADDR:                         (unknown)
GENERAL.MTU:                            0
GENERAL.STATE:                          30 (disconnected)
GENERAL.CONNECTION:                     --
GENERAL.CON-PATH:                       --

GENERAL.DEVICE:                         balena0
GENERAL.TYPE:                           bridge
GENERAL.HWADDR:                         02:42:F1:A7:53:36
GENERAL.MTU:                            1500
GENERAL.STATE:                          10 (unmanaged)
GENERAL.CONNECTION:                     --
GENERAL.CON-PATH:                       --
IP4.ADDRESS[1]:                         10.114.101.1/24
IP4.GATEWAY:                            --
IP4.ROUTE[1]:                           dst = 10.114.101.0/24, nh = 0.0.0.0, mt = 0
IP6.GATEWAY:                            --

GENERAL.DEVICE:                         br-859fdb826d9e
GENERAL.TYPE:                           bridge
GENERAL.HWADDR:                         02:42:5E:B4:3A:53
GENERAL.MTU:                            1500
GENERAL.STATE:                          10 (unmanaged)
GENERAL.CONNECTION:                     --
GENERAL.CON-PATH:                       --
IP4.ADDRESS[1]:                         172.17.0.1/16
IP4.GATEWAY:                            --
IP4.ROUTE[1]:                           dst = 172.17.0.0/16, nh = 0.0.0.0, mt = 0
IP6.GATEWAY:                            --

GENERAL.DEVICE:                         resin-dns
GENERAL.TYPE:                           bridge
GENERAL.HWADDR:                         DE:8B:69:85:D8:38
GENERAL.MTU:                            1500
GENERAL.STATE:                          10 (unmanaged)
GENERAL.CONNECTION:                     --
GENERAL.CON-PATH:                       --
IP4.ADDRESS[1]:                         10.114.102.1/24
IP4.GATEWAY:                            --
IP4.ROUTE[1]:                           dst = 10.114.102.0/24, nh = 0.0.0.0, mt = 0
IP6.GATEWAY:                            --

GENERAL.DEVICE:                         lo
GENERAL.TYPE:                           loopback
GENERAL.HWADDR:                         00:00:00:00:00:00
GENERAL.MTU:                            65536
GENERAL.STATE:                          10 (unmanaged)
GENERAL.CONNECTION:                     --
GENERAL.CON-PATH:                       --
IP4.ADDRESS[1]:                         127.0.0.1/8
IP4.GATEWAY:                            --
IP6.ADDRESS[1]:                         ::1/128
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = ::1/128, nh = ::, mt = 256

GENERAL.DEVICE:                         resin-vpn
GENERAL.TYPE:                           tun
GENERAL.HWADDR:                         (unknown)
GENERAL.MTU:                            1500
GENERAL.STATE:                          10 (unmanaged)
GENERAL.CONNECTION:                     --
GENERAL.CON-PATH:                       --
IP4.ADDRESS[1]:                         10.240.104.11/32
IP4.GATEWAY:                            --
IP4.ROUTE[1]:                           dst = 52.4.252.97/32, nh = 0.0.0.0, mt = 0
IP6.ADDRESS[1]:                         fe80::8d39:ed34:2d6e:9c04/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = ff00::/8, nh = ::, mt = 256, table=255
IP6.ROUTE[2]:                           dst = fe80::/64, nh = ::, mt = 256

The saga continues…

Tried two different sims (one from Twilio, which worked in the past and one from IoTDataworks - both use Tmobile). Also, tried a completely different machine (fin3) with a completely different modem (non sixfab, just the mpcie quectel ec25-g) and oddly I’m getting the same stuck in registered problem from mmcli.

Utterly baffled that now two devices with two different modems and two different sims (9 variants tried) would all fail the same way, especially when one worked in the past.

I did some more digging and found this is happening:

Mar 17 23:10:52 59b8278 ModemManager[741]: [/dev/cdc-wdm0] received message...
                                           <<<<<< RAW:
                                           <<<<<<   length = 25
                                           <<<<<<   data   = 01:18:00:80:03:03:02:83:00:20:00:0C:00:02:04:00:00:00:00:00:01:02:00:C1:08
Mar 17 23:10:52 59b8278 ModemManager[741]: [/dev/cdc-wdm0] received generic response (translated)...
                                           <<<<<< QMUX:
                                           <<<<<<   length  = 24
                                           <<<<<<   flags   = 0x80
                                           <<<<<<   service = "nas"
                                           <<<<<<   client  = 3
                                           <<<<<< QMI:
                                           <<<<<<   flags       = "response"
                                           <<<<<<   transaction = 131
                                           <<<<<<   tlv_length  = 12
                                           <<<<<<   message     = "Get Signal Strength" (0x0020)
                                           <<<<<< TLV:
                                           <<<<<<   type       = "Result" (0x02)
                                           <<<<<<   length     = 4
                                           <<<<<<   value      = 00:00:00:00
                                           <<<<<<   translated = SUCCESS
                                           <<<<<< TLV:
                                           <<<<<<   type       = "Signal Strength" (0x01)
                                           <<<<<<   length     = 2
                                           <<<<<<   value      = C1:08
                                           <<<<<<   translated = [ strength = '-63' radio_interface = 'lte' ]
Mar 17 23:10:52 59b8278 ModemManager[741]: Signal strength (lte): -63 dBm
Mar 17 23:10:52 59b8278 ModemManager[741]: Signal strength: -63 dBm --> 81%
Mar 17 23:10:52 59b8278 ModemManager[741]: Modem /org/freedesktop/ModemManager1/Modem/0: signal quality updated (81)
Mar 17 23:10:52 59b8278 ModemManager[741]: Periodic signal quality checks scheduled in 30s
Mar 17 23:11:06 59b8278 ModemManager[741]: (net/veth4ed2d80): adding device at sysfs path: /sys/devices/virtual/net/veth4ed2d80
Mar 17 23:11:06 59b8278 ModemManager[741]: [filter] (net/veth4ed2d80) port filtered: virtual device
Mar 17 23:11:06 59b8278 ModemManager[741]: (net/veth4ed2d80): adding device at sysfs path: /sys/devices/virtual/net/veth4ed2d80
Mar 17 23:11:06 59b8278 ModemManager[741]: [filter] (net/veth4ed2d80) port filtered: virtual device
Mar 17 23:11:06 59b8278 ModemManager[741]: (net/veth1a80303): adding device at sysfs path: /sys/devices/virtual/net/veth1a80303
Mar 17 23:11:06 59b8278 ModemManager[741]: [filter] (net/veth1a80303) port filtered: virtual device
Mar 17 23:11:06 59b8278 ModemManager[741]: (net/vethf26f229): adding device at sysfs path: /sys/devices/virtual/net/vethf26f229
Mar 17 23:11:06 59b8278 ModemManager[741]: [filter] (net/vethf26f229) port filtered: virtual device
Mar 17 23:11:22 59b8278 ModemManager[741]: loading signal quality...

Plenty of signal quality, but the system seems to keep reassigning the modem to different veths within seconds.
Has anyone seen this before??

Hi, reading through the above thread I have some follow up questions:

  • Have you tested the SIM card on a mobile phone, and made sure it is able to register and work on the carrier’s network?
  • Does that SIM card work on Raspbian on your RPI4 + Hat combination?

If the answer to both the above is yes, I suggest you configure NetwotkManager debug logging with:

nmcli general logging level DEBUG domain ALL

And capture connection logs from both Raspbian and BalenaOS. If you send those logs to us we can then help you debug this further.