balenafin with cellular locks up and goes offline

balenaFin
1

I put a cellular modem on the v1.1.1 fin and it does work at the beginning fine but after a couple of hours it goes offline and never comes up.

After I go and open my enclosure i see that 3V, 5V LEDS are on and the ACT led is blinking. Only a power reset will make it work again

i am using telit LE910-NA V2 module with a soracom simcard.

2021-07-21 10_11_42-SORACOM User Console - Brave
2021-07-21 10_11_42-SORACOM User Console - Brave1131×529 31.1 KB

as you can see the data activity stopped at 9pm yesterday and it never came back online.
balena log cellular.txt (34.3 KB)
here is the log not from that time because i cant get to it right now.

but over that period of time i push some updates to check how it works downloading updates to the fin and the same behavior was occurring. going offline and never coming back.

the fin has power, and soracom console show the modem ONLINE as if it was connected to the network.

has anyone experience such intermittent issues using cellular connectivity?

2

Hey there, can you enable persistent logging on the device so we can collect logs after it’s been rebooted? Specifically it would help to have kernel (dmesg) logs from the time when the modem is disconnected.

Can you connect one of those modems to a device that also has wifi or ethernet? It would be helpful to troubleshoot while the device is online.

Also can you share your version of BalenaOS?

4

let me check if i can get someone to climb the ladder open the enclosure and do a power reset hehe

5

@klutchell how do i enable persisten logging, trying to find out on the documentation but i could not find it.

thanks

6

Oops, I meant to include instructions. You can enable persistent logging on the device dashboard under Device Configuration.

7

@klutchell the device just cameback by it self… dont know how lol

but i did enable the persisten logging now…

do i chieck the logs with this command right
dmesg | tail -n 100 ?

8

persistenlog.txt (36.6 KB)

it shows some errors on the cellular modem… take a look at the file

9

Hi,

I’m seeing multiple USB disconnects/reconnects for that modem.
Is this something you did manually, or does it just lose connection by itself?

If it’s something it did by itself, this could indicate a reset of the modem triggered by software software (either explicit, or some kind of watchdog) or possibly a hardware failure (ex. power).

If you can login to HostOS, you may be able to get more logging through journalctl -am --no-pager.

I also remember seeing an article somewhere a while ago dealing with losing cellular connection on a raspberry pi after hours of it working properly; I will see if I can find that article.

Edit: I found this thread on the ModemManager mailing list, but haven’t found the article I was thinking of just yet.

Edit 2: here someone had an issue that was resolved by updating their OS.
This one deals with a ppp configuration that fixes their issue.

Can you share your OS and modem/connection setup/configuration?
This may help in finding the issue.

2 Likes
12

Thank you for your answer @TJvV

@jorgesea is this now up&running on your side?

14

I need to go and pick it up to where it is connected i have not been able to get there this past week.

it has been offline for 5 days now. but I bet the balenfin is ON, will report once i get it.

16

finally had some time to come back and work on this and i changed the modem to a telit LE910C1-NF and getting this error
couldn’t start network: QMI protocol error (14): ‘CallFailed’

Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] simple connect started...
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] simple connect state (4/8): wait to get fully enabled
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] simple connect state (5/8): register
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] simple connect state (6/8): bearer
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] simple connect state (7/8): connect
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] state changed (registered -> connecting)
Aug 21 15:08:25 dfbe2a2 NetworkManager[1626]: <info>  [1629558505.5190] modem["cdc-wdm0"]: modem state changed, 'registered' --> 'connecting' (reason: user-requested)
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0/bearer133] couldn't start network: QMI protocol error (14): 'CallFailed'
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0/bearer133] call end reason (1): generic-unspecified
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0/bearer133] verbose call end reason (6,55): [3gpp] multiple-connection-to-same-pdn-not-allowed
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0/bearer133] connection attempt #1 failed: QMI protocol error (14): 'CallFailed'
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] state changed (connecting -> registered)
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0/bearer133] connection #1 finished: duration 0s, tx: 0 bytes, rx :0 bytes
Aug 21 15:08:25 dfbe2a2 NetworkManager[1626]: <info>  [1629558505.7177] modem["cdc-wdm0"]: modem state changed, 'connecting' --> 'registered' (reason: user-requested)
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] simple connect started...
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] simple connect state (4/8): wait to get fully enabled
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] simple connect state (5/8): register
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] simple connect state (6/8): bearer
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] simple connect state (7/8): connect
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] state changed (registered -> connecting)
Aug 21 15:08:25 dfbe2a2 NetworkManager[1626]: <info>  [1629558505.7254] modem["cdc-wdm0"]: modem state changed, 'registered' --> 'connecting' (reason: user-requested)
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0/bearer134] couldn't start network: QMI protocol error (14): 'CallFailed'
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0/bearer134] call end reason (1): generic-unspecified
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0/bearer134] verbose call end reason (6,55): [3gpp] multiple-connection-to-same-pdn-not-allowed
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0/bearer134] connection attempt #1 failed: QMI protocol error (14): 'CallFailed'
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0] state changed (connecting -> registered)
Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0/bearer134] connection #1 finished: duration 0s, tx: 0 bytes, rx :0 bytes
Aug 21 15:08:25 dfbe2a2 NetworkManager[1626]: <info>  [1629558505.8541] modem["cdc-wdm0"]: modem state changed, 'connecting' --> 'registered' (reason: user-requested)
Aug 21 15:08:25 dfbe2a2 NetworkManager[1626]: <warn>  [1629558505.8544] modem-broadband[cdc-wdm0]: failed to connect modem: QMI protocol error (14): 'CallFailed'
Aug 21 15:08:25 dfbe2a2 NetworkManager[1626]: <info>  [1629558505.8545] device (cdc-wdm0): state change: prepare -> failed (reason 'unknown', sys-iface-state: 'managed')````
17

Hi,

The interesting bit here is the line

Aug 21 15:08:25 dfbe2a2 ModemManager[1451]: [modem0/bearer134] verbose call end reason (6,55): [3gpp] multiple-connection-to-same-pdn-not-allowed

I’ve not seen this error before, but a quick google pointed me to this stackoverflow question, where apparently the way to reproduce it is to first (try to) connect to an incorrect APN, followed by the correct APN.

From what I’ve gathered so far, it seems like your modem or SIM card doesn’t like you opening a second connection.
When searching this error, I also found multiple threads about having multiple network interfaces with the same subnet.

Can you share your network configuration?

18

Well a note on this… the cellular module i have has also a slot for a simcard and well the BalenaFin also has a slot for a simcard…

could it be related, should we tell it what to use?

19

Hi @jorgesea ,

If the modem does have an internal SIM slot, you should be using that one. Most of the times, when the modem has an internal SIM slot, the SIM lines are not connected to the mpcie slot. The SIM slot on the Fin is essentially disconnected from the modem.

Also, could you paste the output of dmesg when the modem disconnects?

Regards,

21

Hello @jorgesea

did you try what my colleague suggested? Let us know if you could solve your issue!

Let’s stay connected