@krenom and @jha, let us know the result of your experiments if you can. If the 100MB data limit is something configured by the SIM provider, then of course there isn’t much we could do in the balenaOS / ModemManager side. Thanks!
Good evening!
A solid week of running has proved that the 100MB limit was the issue. The new SIM that isn’t limited hasn’t killed a session once. It has remained online and happily throwing our data around without a problem. Several of our containers have shown issues, but that’s a different story and nothing to do with the connectivity.
So, we’re not too sure of the exact procedure needed to manually start a new session (though they said that the SIM would automatically start up a new session for us) - but a very brief attempt was made at restarting the modem if it failed to connect for any reason.
Using mmcli, restarting the modem is as easy as an mmcli -m 0 -r or something rather similar, and worked well. Once the modem came back up and reconnected, it did so with a new session (presumably, as it would start working again).
Using DBus in our networking container, hitting the modem interface with a reset just seemed to kill off the device as a whole and a reboot was needed to have it come back up. So that was pretty strange.
However, now I’m tangenting into something else, slightly.
So, sadly no immediate nice fix for anyone to copy (we sledge-hammered a device reboot in there if it’s been offline for over an hour, though, so there’s that), but can confirm the particular cause of our version of this error and a pretty good fix of simply moving to a different SIM provider.
Using mmcli, restarting the modem is as easy as an
mmcli -m 0 -ror something rather similar, and worked well. […]
Using DBus in our networking container, hitting the modem interface with a reset just seemed to kill off the device as a whole and a reboot was needed to have it come back up.
Thank you for sharing the results @krenom! Great stuff. On the quote above, for what it is worth, I was looking at source code implementation of mmcli (the beauty of open source) 
and what I found indicates that mmcli -m 0 -r actually ends up calling the same DBUS Reset method you were calling:
mmcli -m 0 -r
→ reset_flag
→ mm_modem_reset
→ mm_gdbus_modem_call_reset_sync
→ DBUS Reset
hi there
I work with @krenom and have picked up the baton on this issue - to make our devices able to recover a GSM connection after a sim deactivation/ re-activation.
Our aim had been to use DBus in our networking container to perform the modem reset when the device gets into this state. As we have dug into trying to make this work we are not so sure the problem is with our dbus implementation. I just wanted to share what we have observed so far…
So when the GSM connection is in this state - querying the modem interface from the host OS looks like this
$nmcli
cdc-wdm0: connecting (prepare) to pod
“cdc-wdm0”
gsm (qcserial, qmi_wwan), hw, mtu 1500
and the state of the connection is NM_CONNECTIVITY_LIMITED
As @krenom says, we had used ModemManager to perform a reset on the host OS at this point and this allows the interface to be reset and the GSM connection will successfully re-establish on the next attempt.
$mmcli -m 0 -r
However, this interface reset only appears to work reliably if the device has some connection at this point (in our dev environment the devices would be connected over ethernet at this point and the command issued from a host OS terminal).
If the device is currently offline - as it would be in production - and we simply schedule this same mm interface reset to take place - it doesn’t work as expected
A quick, direct simulation of this was to issue a delayed reset from the host OS terminal whilst we reactivated the sim
$nohup bash -c ‘sleep 180; mmcli -m 0 -r’ > /dev/null
Executing this command causes our containers to restart. When they come back up, the GSM connection is typically still dead (I say typically because we have observed the connection re-establish on 1 occasion but other times it hasn’t )
The ModemManager logs still appear as per @krenom post on Aug 21
So for us the only foolproof way to re-establish the GSM connection at the moment, is a hard reboot, killing the power to the modem. Anything short of this doesn’t want to play ball 
So for us the only foolproof way to re-establish the GSM connection at the moment, is a hard reboot, killing the power to the modem.
In the unlikely event this comment is useful, the bit about “killing the power to the modem” made me wonder if the following sequence would at least avoid the need to reboot the device:
mmcli -m 0 --set-power-state-off
sleep 1 # no specific reason other than "time heals most wounds?" :-)
mmcli -m 0 --set-power-state-on
Also inspired by this thread:
https://lists.freedesktop.org/archives/modemmanager-devel/2016-January/002522.html
hmm interesting. thanks @pdcastro
yes I will take a look at that. We have now included a ups with our devices so we are able to carry out an unattended hard reboot which means we can use this method instead for the time being.
Hi Iang,
I was wondering if you progressed on the issue you had. Its something that I have seen in production with a few of my devices. My provider had a nationwide outage, which resulted in modemmanager showing only this:
mmcli -m 1
-------------------------
General | dbus path: /org/freedesktop/ModemManager1/Modem/1
-------------------------
System | device: /sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.4
| drivers: qmi_wwan, option1
| plugin: simtech
| primary port: cdc-wdm0
| ports: ttyUSB0 (qcdm), ttyUSB1 (gps), ttyUSB2 (at), ttyUSB3 (at),
| wwan0 (net), ttyUSB4 (audio), cdc-wdm0 (qmi)
-------------------------
Status | signal quality: 0% (cached)
-------------------------
Modes | supported: allowed: any; preferred: none
| current: allowed: any; preferred: none
I’ve never seen modemmanager so confused
I connected a internet dongle and suddenly the modem did fully initialize without reboot.