Lost Dashboard functionality for running device

Product support
1

Have a Device in Admin-Horrormode:
generic_x86_64
HOST OS VERSION: balenaOS 2.83.18+rev1
SUPERVISOR VERSION: 12.10.3

It is lost by dashboard, but running. I’d like to know how to get it back :slight_smile:

What is working:

  • Logs
  • All Services (according to logs)
  • Console for Host-OS (but trying with balena-engine stats or ps makes it crash every time)
  • Online Status Indicator (shows “online” and “heartbeat only” when hostOS console crashes for a short while)
  • NTP-Sync

What is not working:

  • All actions from Dashboard (Restart/Stop Services, Update Supervisor, Update to new Release)
  • Device Status (CPU/Memory/Temperature/Storage) didn’t moved for hours (CPU@93%)

Device health checks:
all green but

Some networking issues detected: 
test_upstream_dns: DNS lookup failed for 0.resinio.pool.ntp.org via upstream: 8.8.8.8
test_upstream_dns: DNS lookup failed for api.balena-cloud.com via upstream: 8.8.8.8

Sometimes also:

test_balena_registry: Could not communicate with registry2.balena-cloud.com for authentication

But:

root@f1e23ab:~# nslookup api.balena-cloud.com
Server:    127.0.0.2
Address 1: 127.0.0.2 f1e23ab

Name:      api.balena-cloud.com
Address 1: 52.73.205.134 ec2-52-73-205-134.compute-1.amazonaws.com
...
Address 9: 2600:1f18:6600:7f00:28b6:a5e0:57b6:8a01
...

root@f1e23ab:~# nslookup 0.resinio.pool.ntp.org
Server:    127.0.0.2
Address 1: 127.0.0.2 f1e23ab

Name:      0.resinio.pool.ntp.org
Address 1: 162.159.200.1 time.cloudflare.com
Address 2: 136.243.202.118 time01.nevondo.com
Address 3: 78.46.204.247 nono.com
Address 4: 217.91.44.17 www.kashra.com

and

root@f1e23ab:~# curl -kv https://registry2.balena-cloud.com
*   Trying 35.174.239.192:443...
* Connected to registry2.balena-cloud.com (35.174.239.192) port 443 (#0)
* found 128 certificates in /etc/ssl/certs/ca-certificates.crt
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
....
* Connection #0 to host registry2.balena-cloud.com left intact

on Firewall i see also a lot of (accepted) outbound TCP/443 and UDP/123 Traffic…

Supervisor State (NOT Target Supervisor State):

{
  "api_port": 48484,
  "ip_address": "192.168.0.20",
  "os_version": "balenaOS 2.83.18+rev1",
  "mac_address": "20:87:...",
  "supervisor_version": "12.11.23", <-- This is target version! According to Dashboard not installed!
  "update_pending": false,
  "update_failed": false,
  "update_downloaded": false,
  "commit": "e2c86f..", <-- This is target version! According to Dashboard not installed!
  "status": "Idle",
  "download_progress": null
}

Device Diagnostics cancels with error

diagnose failed: exit code undefined

Any idea what to do to revive this zombie? Remark: It is geographically far away, so not easy to reboot and so on :slight_smile:

2

Hi @ilgra – we can try to restore the Device to your dashboard, if possible. DM m e the UUID of the device when you get a moment. Thanks!

4

Hi @ilgra,
I think it is no longer needed but I still try to ask you if by any chance you have a way to confirm if the aforementioned device still has problems? Or if, as asked by my colleague, is possible to share the UUID of the device with us so that we can restore it