Huawei E3372 crashes and cellular gets stuck

Product support balenaOS
1

Huawei E3372 4G modem works perfectly for about ~4 hours and then it always gets stuck with message “timed out”, I can see this in the dmesg:

[Mar28 00:15] ------------[ cut here ]------------
[  +0.004719] NETDEV WATCHDOG: wwp1s0u1u1i2 (huawei_cdc_ncm): transmit queue 0 timed out
[  +0.008088] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:461 dev_watchdog+0x2ac/0x2b8
[  +0.008380] Modules linked in: ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 ip6table_filter sg ip6_tables bnep hci_uart btbcm serdev bluetooth ecdh_generic sr_mod cdrom huawei_cdc_ncm option usb_wwan cdc_wdm cdc_ncm usbnet usbserial ipt_MASQUERADE nf_conntrack_netlink nfnetlink br_netfilter xt_owner i2c_dev bcm2835_v4l2(C) bcm2835_mmal_vchiq(C) v4l2_common videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videobuf2_common videodev brcmfmac media brcmutil spidev vc_sm_cma(C) sha256_generic cfg80211 vc4 rfkill snd_soc_core snd_compress snd_bcm2835(C) v3d snd_pcm_dmaengine gpu_sched snd_pcm snd_timer spi_bcm2835 vchiq(C) snd uio_pdrv_genirq uio fixed nls_ascii nls_cp437 vfat fat sch_fq_codel
[  +0.062717] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C        4.19.71 #1
[  +0.007495] Hardware name: Raspberry Pi 4 Model B Rev 1.1 (DT)
[  +0.005912] pstate: 80000005 (Nzcv daif -PAN -UAO)
[  +0.004856] pc : dev_watchdog+0x2ac/0x2b8
[  +0.004060] lr : dev_watchdog+0x2ac/0x2b8
[  +0.004058] sp : ffffff8008003d30
[  +0.003353] x29: ffffff8008003d30 x28: ffffff80095a50c0
[  +0.005385] x27: ffffff8008b06000 x26: 00000000ffffffff
[  +0.005384] x25: 0000000000000140 x24: ffffffc0f158f480
[  +0.005384] x23: ffffffc0f158f45c x22: ffffffc0f6337c80
[  +0.005384] x21: ffffff80095a6000 x20: ffffffc0f158f000
[  +0.005384] x19: 0000000000000000 x18: 0000000000000010
[  +0.005384] x17: 0000000000000000 x16: 0000000000000000
[  +0.005383] x15: ffffffffffffffff x14: 3020657565757120
[  +0.005383] x13: 74696d736e617274 x12: 203a296d636e5f63
[  +0.005384] x11: 64635f6965776175 x10: 6828203269317531
[  +0.005384] x9 : 7530733170777720 x8 : 3a474f4448435441
[  +0.005383] x7 : 572056454454454e x6 : ffffffc0fbf609e0
[  +0.005383] x5 : ffffffc0fbf609e0 x4 : 00000040f29d0000
[  +0.005385] x3 : 0000000000000104 x2 : ffffff800958f018
[  +0.005384] x1 : e31ef0ce273d1d00 x0 : 0000000000000000
[  +0.005384] Call trace:
[  +0.002473]  dev_watchdog+0x2ac/0x2b8
[  +0.003709]  call_timer_fn+0x34/0x1c8
[  +0.003707]  expire_timers+0xc4/0x158
[  +0.003706]  run_timer_softirq+0xb4/0x1a0
[  +0.004059]  __do_softirq+0x180/0x3e8
[  +0.003708]  irq_exit+0xc8/0xe8
[  +0.003179]  __handle_domain_irq+0x9c/0x108
[  +0.004235]  gic_handle_irq+0x58/0xa8
[  +0.003707]  el1_irq+0xb4/0x130
[  +0.003178]  arch_cpu_idle+0x2c/0x238
[  +0.003708]  default_idle_call+0x38/0x44
[  +0.003973]  do_idle+0x240/0x258
[  +0.003266]  cpu_startup_entry+0x28/0x30
[  +0.003972]  rest_init+0xd0/0xdc
[  +0.003268]  start_kernel+0x43c/0x464
[  +0.003706] ---[ end trace 3288d83db010eff7 ]---

I’m using following config:

[connection]
id=cellular
type=gsm
autoconnect=true

[gsm]
apn=internet

I think these cellular sticks are notoriously bad, so I probably need a way to restart the device. But this seems to get stuck everytime in few hours, so there must be something else too.

Other logs when it gets stuck: (Device disappears):

root@fec0fc3:~# nmcli c
NAME                UUID                                  TYPE      DEVICE
Wired connection 1  5055354e-3f88-396b-86d6-4e65a91cf527  ethernet  eth0
supervisor0         3a1f0637-8f82-412c-a23a-c6d62932d09b  bridge    supervisor0
cellular            171737db-9c69-39bd-8cf6-459edd25039c  gsm       --

Notice the “–” in the table.

2

Hi,
Thanks for reporting this.

Would you clarify what board you are working with and what balenaOS version you are running?

6

Unfortunately, that is a symptom of a modem firmware crash. The netdev kernel layer is detecting that the wwan network interface exposed by the modem went silent. There is nothing we can do from system userspace or even from the kernel, the “fix” for this situation involves a full modem power cycle (e.g. unplugging/replugging)

1 Like
9

Hi, we would love to investigate why the device gets stuck. Could you share the following:

  • balenaOS version
  • The board you are using
11

Hello!

Balena OS version is: 2.46.1 rev3.
Raspberry PI 4.

I also noticed that it’s always preceded by these two lines:

[Fri Apr 10 17:37:53 2020] xhci_hcd 0000:01:00.0: WARN Cannot submit Set TR Deq Ptr
[Fri Apr 10 17:37:53 2020] xhci_hcd 0000:01:00.0: A Set TR Deq Ptr command is pending.
[Fri Apr 10 17:38:51 2020] ------------[ cut here ]------------
[Fri Apr 10 17:38:51 2020] NETDEV WATCHDOG: wwp1s0u1u1i2 (huawei_cdc_ncm): transmit queue 0 timed out
...

It does start to work again if I unplug and plug it back (e.g. with uhubctl).

14

@Ciantic yes from what you describe, it definitely sounds like firmware on the modem crashes as @aleksander0m suggests. That is why it works again when you replug it as it re-initialises your modem firmware. I think you might need to work around this by resetting the usb hub when it fails as there probably isn’t a lot we can do from the modemManager/OS side.

1 Like