Lost Dashboard functionality for running device

Have a Device in Admin-Horrormode:
generic_x86_64
HOST OS VERSION: balenaOS 2.83.18+rev1
SUPERVISOR VERSION: 12.10.3

It is lost by dashboard, but running. I’d like to know how to get it back :slight_smile:

What is working:

  • Logs
  • All Services (according to logs)
  • Console for Host-OS (but trying with balena-engine stats or ps makes it crash every time)
  • Online Status Indicator (shows “online” and “heartbeat only” when hostOS console crashes for a short while)
  • NTP-Sync

What is not working:

  • All actions from Dashboard (Restart/Stop Services, Update Supervisor, Update to new Release)
  • Device Status (CPU/Memory/Temperature/Storage) didn’t moved for hours (CPU@93%)

Device health checks:
all green but

Some networking issues detected: 
test_upstream_dns: DNS lookup failed for 0.resinio.pool.ntp.org via upstream: 8.8.8.8
test_upstream_dns: DNS lookup failed for api.balena-cloud.com via upstream: 8.8.8.8

Sometimes also:

test_balena_registry: Could not communicate with registry2.balena-cloud.com for authentication

But:

root@f1e23ab:~# nslookup api.balena-cloud.com
Server:    127.0.0.2
Address 1: 127.0.0.2 f1e23ab

Name:      api.balena-cloud.com
Address 1: 52.73.205.134 ec2-52-73-205-134.compute-1.amazonaws.com
...
Address 9: 2600:1f18:6600:7f00:28b6:a5e0:57b6:8a01
...

root@f1e23ab:~# nslookup 0.resinio.pool.ntp.org
Server:    127.0.0.2
Address 1: 127.0.0.2 f1e23ab

Name:      0.resinio.pool.ntp.org
Address 1: 162.159.200.1 time.cloudflare.com
Address 2: 136.243.202.118 time01.nevondo.com
Address 3: 78.46.204.247 nono.com
Address 4: 217.91.44.17 www.kashra.com

and

root@f1e23ab:~# curl -kv https://registry2.balena-cloud.com
*   Trying 35.174.239.192:443...
* Connected to registry2.balena-cloud.com (35.174.239.192) port 443 (#0)
* found 128 certificates in /etc/ssl/certs/ca-certificates.crt
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
....
* Connection #0 to host registry2.balena-cloud.com left intact

on Firewall i see also a lot of (accepted) outbound TCP/443 and UDP/123 Traffic…

Supervisor State (NOT Target Supervisor State):

{
  "api_port": 48484,
  "ip_address": "192.168.0.20",
  "os_version": "balenaOS 2.83.18+rev1",
  "mac_address": "20:87:...",
  "supervisor_version": "12.11.23", <-- This is target version! According to Dashboard not installed!
  "update_pending": false,
  "update_failed": false,
  "update_downloaded": false,
  "commit": "e2c86f..", <-- This is target version! According to Dashboard not installed!
  "status": "Idle",
  "download_progress": null
}

Device Diagnostics cancels with error

diagnose failed: exit code undefined

Any idea what to do to revive this zombie? Remark: It is geographically far away, so not easy to reboot and so on :slight_smile:

Hi @ilgra – we can try to restore the Device to your dashboard, if possible. DM m e the UUID of the device when you get a moment. Thanks!