Error: Could not create NMClient object: Timeout was reached.

Product support
1

We frequently see devices in the field where nmcli does not work:

root@e448591:~# nmcli c show
Error: Could not create NMClient object: Timeout was reached.

However, the NetworkManager service seems to be happily running:

root@e448591:~# systemctl status NetworkManager
● NetworkManager.service - Network Manager
   Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/NetworkManager.service.d
           └─NetworkManager.conf
   Active: active (running) since Thu 2020-05-28 17:33:02 UTC; 1min 13s ago
     Docs: man:NetworkManager(8)
 Main PID: 10939 (NetworkManager)
   CGroup: /system.slice/NetworkManager.service
           └─10939 /usr/sbin/NetworkManager --no-daemon

The diagnostics indicate issues with the container engine although it’s up so it seems to have recovered.

Googling does not yield much

We see this problem somewhat often. Rebooting often solves the issue but we’re a bit gun-shy as we’ve had incidences where the device has not returned from reboots, requiring manual reset, which is an issue since our devices are remote.

e44859117a15b0189c30c6b7c03ca604_2020.05.28_17.59.41+0000.log (899.3 KB)

Screen Shot 2020-05-28 at 11.35.29 AM
Screen Shot 2020-05-28 at 11.35.29 AM1588×750 72.2 KB

4

Hi there, can you let us know what balenaOS version you are using? We found https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/404, which might be related to what you are seeing, but we need to investigate further.

6

We’re running 2.48. I’m seeing the issue right now on 1e464992d4426c665b1a7c45857ecc40. I have provided support access, but I may have to reboot to avoid the system going offline completely.

It could be related to the issue you linked to. We have a script that periodically tears down the wifi and brings it back up again as a response to a problem we have seen. It may be that this process is what causes the issue. Also, our internet connection is via cellular and are not very reliable.

7

Thanks for the information! I’ll ping our OS team to see if they have any pointers.

15

Hi,
I’ve just checked the device and see nmcli working normally there.

We suggest editing the network manager configuration to disable connectivity check and observing the device to see if it has any effect.
The connectivity check settings are defined in /etc/NetworkManager/conf.d/os-networkmanager.conf.
Normally, it’s controlled with config.json:


And will be rewritten on reboot. But if you need to do an experiment without a reboot, you can edit the file directly.

Also, it’s worth checking the difference between NetworkManager/nmcli installed on the host OS and in your container. Big version differences can cause problems as well.

18

Thanks Roman,

The versions are 1.20.8 and 1.120.2 so presumably close. I’ll continue playing around to see if changing the connectivity checks affects stability. As it is now, the resident NetworkManager connectivity check is every hour.

19

Thanks for the update. Please let us know if you need any further help after you resume your tests.
Georgia