Hi,
We are having an issue where we now have multiple devices failing to report their initial state to the Balena Cloud and are always appearing as Online(VPN Only) in the dashboard. The device passes all of it’s diagnostic checks run via the dashboard.
It does not appear to be a dns server, as I am able to ping the API endpoint properly. I’m out of ideas as to what may be going on. Any guidance would be appreciated. Thank you!
curl https://api.balena-cloud.com/ping returns OK consistantly
Logs:
balena-logs.txt (37.8 KB)
Device is a Balena Fin CM3 running balenaOS 2.83.21+rev1 and supervisor 12.10.3
Hey there, I see the supervisor is reporting a number of ETIMEDOUT errors.
Oct 21 16:44:50 676e059 f2b562d0c5dd[1560]: [error] Error reporting initial configuration, will retry Error: ETIMEDOUT
Oct 21 16:44:50 676e059 f2b562d0c5dd[1560]: [error] at Timeout.<anonymous> (/usr/src/app/dist/app.js:10:1666140)
Oct 21 16:44:50 676e059 f2b562d0c5dd[1560]: [error] at listOnTimeout (internal/timers.js:549:17)
Oct 21 16:44:50 676e059 f2b562d0c5dd[1560]: [error] at processTimers (internal/timers.js:492:7)
Oct 21 16:45:50 676e059 f2b562d0c5dd[1560]: [error] Error reporting initial configuration, will retry Error: ETIMEDOUT
Oct 21 16:45:50 676e059 f2b562d0c5dd[1560]: [error] at Timeout.<anonymous> (/usr/src/app/dist/app.js:10:1666140)
Oct 21 16:45:50 676e059 f2b562d0c5dd[1560]: [error] at listOnTimeout (internal/timers.js:549:17)
Oct 21 16:45:50 676e059 f2b562d0c5dd[1560]: [error] at processTimers (internal/timers.js:492:7)
Oct 21 16:46:51 676e059 f2b562d0c5dd[1560]: [error] Error reporting initial configuration, will retry Error: ETIMEDOUT
Oct 21 16:46:51 676e059 f2b562d0c5dd[1560]: [error] at Timeout.<anonymous> (/usr/src/app/dist/app.js:10:1666140)
Oct 21 16:46:51 676e059 f2b562d0c5dd[1560]: [error] at listOnTimeout (internal/timers.js:549:17)
Oct 21 16:46:51 676e059 f2b562d0c5dd[1560]: [error] at processTimers (internal/timers.js:492:7)
Were these devices online previously? Do they have anything environmental in common, like the same physical location or firewall?
Do you know if your ISP has provided IPv6 addresses, or IPv4, or both? Are you able to connect to both of these endpoints?
curl -I https://ipv4.google.com
curl -I https://ipv6.google.com
Thanks for the quick response!
Yes, these devices were online previously at this location.
They are all in the same physical location, but there is no firewall or anything like that setup just a standard wifi router from the isp. We have devices working in other locations.
Looks like we don’t have IPv6 support. Would that cause an issue?
Looks like we don’t have IPv6 support. Would that cause an issue?
Only if your gateway or ISP is assigning an IPv6 address, but failing to actually route it.
Are these the only two devices at that location, and also the only two experiencing this issue?
Would you be willing to try disabling IPv6 on one of those devices as a test? If you have VPN access you should be able to run these commands in a host shell.
$ nmcli con show
NAME UUID TYPE DEVICE
Wired connection 087fd7cd-9a6b-423c-8b64-c30cbf6a7126 ethernet eno1
...
$ nmcli con edit 087fd7cd-9a6b-423c-8b64-c30cbf6a7126
nmcli> set ipv6.method disable
nmcli> save persistent
Connection 'Wired connection' (087fd7cd-9a6b-423c-8b64-c30cbf6a7126) successfully updated.
nmcli> quit
$ nmcli con down 087fd7cd-9a6b-423c-8b64-c30cbf6a7126; nmcli con up 087fd7cd-9a6b-423c-8b64-c30cbf6a7126
If these steps resolve the issue for one device, feel free to apply it to the other device as well.
We’ve seen recently that some devices will be assigned IPv6 addresses by the gateway or ISP but be unable to actually route the traffic. We have some workarounds planned for the OS to try both IPv4 and IPv6 if both are present, but currently each service may choose a different route and it’s possible that in this case your supervisor is preferring IPv6.
That’s the theory anyway, let us know how it goes!