Device Keeps dropping Connection

Hi,

We have a device running remotely that connects to an industrial 3G/Wifi Router. However, our device appears to be losing its connection. It goes between being completely offline, then connects with VPN only.

I tried to run a device Health Check but it went offline before it could complete.

I can provide support access if someone would be able to have a look?

Thanks,
Gregor

I had a look in the Journalctl for logs about the supervisor and found the following


It appears there is an empty error report and then the health check is failing.

I have activated persistent logging in the hope that this will allow more insight into what is happening to cause the device to be stuck between offline and VPN only.

Hello @GregorR1

Could you please tell us more details about the idunstrial 3g/wifi router you are using and your device type?

Thanks

Hi @mpous,

We are using a Teltonika RUT230 and our balena device is a RPi4. The Teltonika is creating a Wifi AP, which our RPi4 is connecting to.

We have an identical setup with another one of our devices, which has been running without any issues.

Last night I tried to update the supervisor, but nothing appeared to happen. The device continued to be offline/VPN only overnight.

This morning, the device is back online and transmitting to our MQTT server and the supervisor update has been completed.

I can enable support access if this would be helpful to find out what went wrong? But for now, it appears the issue is solved.

Yes @GregorR1 please! grant us support access and let us know your device ID to access to it

Hi I have granted the support access, the device UUID is a452f410290cb0a129c2201f7642c340

Thanks,
Gregor

Hi,

Just an update, the device appears to have gone back into the error state and is currently connected VPN Only.

I will grant support access again so you can have a look. I already enabled persistent logging when the device was having issues the first time so hopefully, we will be able to see what has gone wrong.

Thanks,
Gregor

ok! I remember checking the device when it was online and everything was correct. Ping me again when you will get the device online. Could you please confirm that the modem has always connectivity?

Hi @mpous,

As far as I can tell from the balena dashboard the device has been “Online” for a day, so it would appear that the internet connection is reasonably stable.

When I was trying to debug this issue last week I noticed that the device would be stable Online (VPN Only) until I tried to access the services through either Public URL or SSH. This would cause it to go offline for a bit then appear Online (VPN only) shortly afterwards.

While it is Online (VPN only) I am not receiving any data from it, so it is as if my containers are not running. I can still access the Host OS through SSH, so my gut feeling is it has something to do with the Supervisor or my containers as opposed to a network issue.

I think I will have to flash a new sd card and have it taken out to the site as the device is still not working properly.

I will get the sd card brought back to the office so I can set up a test RPi4 to try and figure out what went wrong.

Hello @GregorR1 do you have more devices connected to the same router? do they work normally?

On your logs we catcher Event: Device state report failure {"error":"ESOCKETTIMEDOUT"} so it looks like there are connectivity issues.

Hi @mpous,

Unfortunately, we only have one device connected to that router.

I will get the new card taken to the site, and hopefully, this will be stable enough to let us debug any network issues. I will get the engineer to also plug the device into the router via ethernet in case there is an unstable WiFi connection.

Thanks,
Gregor

ok @GregorR1 keep us updated!