How to debug offline GSM device

Hello,

I am testing balena with a device that has an LTE internet connection via a Huawei 909 modem. It’s getting offline sometimes (blinking 4 times in a row) and I have trouble finding out why. Can you give me some hints on how to debug? For example, how would I connect to the device to retrieve logs? My understanding is that when I reboot, the logs are gone, right?

Also it seems when the device is gone offline, connecting it to an Internet router via LAN does not help, it stays offline.

Any help much appreciated.
Bruno

Yes, when rebooting a device, the logs are gone unless you enable persistent logging in device configuration (on web dashboard or in config.json). You can use dev image to connect to device with ssh without password: https://www.balena.io/docs/reference/OS/overview/2.x/#dev-vs-prod-images

Then you can run journalctl -u ModemManager --no-pager to get network related logs. You can use serial cable or ethernet cable to connect to device when it’s running dev image.

Thank you. I managed to get the device online by connecting to my router via lan. This is what the command line you suggested brings up:

root@9d52738:~# journalctl -u ModemManager --no-pager
-- Logs begin at Wed 2019-11-27 09:07:10 UTC, end at Wed 2019-11-27 09:33:51 UTC. --
-- No entries --

This is what the NetworkManager logs:

-- Logs begin at Wed 2019-11-27 09:09:20 UTC, end at Wed 2019-11-27 09:41:11 UTC. --
Nov 27 09:09:41 9d52738 NetworkManager[791]: <info>  [1574845781.9134] manager: (veth98abddc): new Veth device (/org/freedesktop/NetworkManager/Devices/2467)
Nov 27 09:09:41 9d52738 NetworkManager[791]: <info>  [1574845781.9704] device (veth0704b96): released from master device br-4a4858757201
Nov 27 09:09:42 9d52738 NetworkManager[791]: <info>  [1574845782.3362] manager: (vetha2ddd6b): new Veth device (/org/freedesktop/NetworkManager/Devices/2468)
Nov 27 09:09:42 9d52738 NetworkManager[791]: <info>  [1574845782.3450] manager: (veth6a034d5): new Veth device (/org/freedesktop/NetworkManager/Devices/2469)
Nov 27 09:09:43 9d52738 NetworkManager[791]: <info>  [1574845783.7902] device (veth6a034d5): carrier: link connected
Nov 27 09:10:11 9d52738 NetworkManager[791]: <info>  [1574845811.9920] manager: (vetha2ddd6b): new Veth device (/org/freedesktop/NetworkManager/Devices/2470)
Nov 27 09:10:12 9d52738 NetworkManager[791]: <info>  [1574845812.0838] device (veth6a034d5): released from master device br-4a4858757201
Nov 27 09:10:12 9d52738 NetworkManager[791]: <info>  [1574845812.3754] manager: (veth863873f): new Veth device (/org/freedesktop/NetworkManager/Devices/2471)
Nov 27 09:10:12 9d52738 NetworkManager[791]: <info>  [1574845812.4221] manager: (vethd1429ce): new Veth device (/org/freedesktop/NetworkManager/Devices/2472)
Nov 27 09:10:13 9d52738 NetworkManager[791]: <info>  [1574845813.8706] device (vethd1429ce): carrier: link connected
Nov 27 09:10:42 9d52738 NetworkManager[791]: <info>  [1574845842.3340] manager: (veth863873f): new Veth device (/org/freedesktop/NetworkManager/Devices/2473)
Nov 27 09:10:43 9d52738 NetworkManager[791]: <info>  [1574845843.0175] device (vethd1429ce): released from master device br-4a4858757201
Nov 27 09:10:43 9d52738 NetworkManager[791]: <info>  [1574845843.3576] manager: (vethb0cf9af): new Veth device (/org/freedesktop/NetworkManager/Devices/2474)
...
Nov 27 09:24:10 9d52738 NetworkManager[791]: <info>  [1574846650.0000] manager: NetworkManager state is now CONNECTED_SITE
Nov 27 09:24:10 9d52738 NetworkManager[791]: <info>  [1574846650.0005] policy: set 'Wired connection 1' (eth0) as default for IPv4 routing and DNS
Nov 27 09:24:10 9d52738 NetworkManager[791]: <info>  [1574846650.0028] dns-mgr: Writing DNS information to /sbin/resolvconf
Nov 27 09:24:10 9d52738 NetworkManager[791]: <info>  [1574846650.0575] device (eth0): Activation: successful, device activated.
Nov 27 09:24:10 9d52738 NetworkManager[791]: <info>  [1574846650.6233] manager: NetworkManager state is now CONNECTED_GLOBAL
Nov 27 09:24:12 9d52738 NetworkManager[791]: <info>  [1574846652.6655] manager: (vethb75b3cd): new Veth device (/org/freedesktop/NetworkManager/Devices/2554)
Nov 27 09:24:12 9d52738 NetworkManager[791]: <info>  [1574846652.7210] device (veth3e54357): released from master device br-4a4858757201
Nov 27 09:24:13 9d52738 NetworkManager[791]: <info>  [1574846653.0739] manager: (veth3647363): new Veth device (/org/freedesktop/NetworkManager/Devices/2555)
Nov 27 09:24:13 9d52738 NetworkManager[791]: <info>  [1574846653.0801] manager: (veth2d64bda): new Veth device (/org/freedesktop/NetworkManager/Devices/2556)
Nov 27 09:24:14 9d52738 NetworkManager[791]: <info>  [1574846654.6089] device (veth2d64bda): carrier: link connected
Nov 27 09:24:45 9d52738 NetworkManager[791]: <info>  [1574846685.4441] manager: (resin-vpn): new Tun device (/org/freedesktop/NetworkManager/Devices/2557)

I ask myself, why I do not get any logs before 9:09. I think I lost the LTE connection a couple hours ago. Any idea?

Device may have rebooted and flushed the journal. journalctl --list-boots will show what time the device rebooted. You can get additional logs with dmesg (https://www.balena.io/docs/learn/manage/ssh-access/#dmesg). The best thing to do now would be to apply persistent logging and reboot the device to allow us to get more information in the future.

I will apply persistent logging.

But what does the following tell me?

root@9d52738:~# journalctl --list-boots
 0 f582cd846ffe47b1b88b131fb8f32cfb Wed 2019-11-27 09:16:15 UTC—Wed 2019-11-27 10:04:31 UTC
root@9d52738:~# uptime
 10:05:00  up 2 days 17:11,  1 user,  load average: 0.82, 0.70, 0.50

The logs might have been cleaned up. We can debug further if this problem reoccurs thanks to the persistent logging. Not sure if we can do anything else for now.