Lost Dashboard functionality for running device

Have a Device in Admin-Horrormode:
HOST OS VERSION: balenaOS 2.83.18+rev1

It is lost by dashboard, but running. I’d like to know how to get it back :slight_smile:

What is working:

  • Logs
  • All Services (according to logs)
  • Console for Host-OS (but trying with balena-engine stats or ps makes it crash every time)
  • Online Status Indicator (shows “online” and “heartbeat only” when hostOS console crashes for a short while)
  • NTP-Sync

What is not working:

  • All actions from Dashboard (Restart/Stop Services, Update Supervisor, Update to new Release)
  • Device Status (CPU/Memory/Temperature/Storage) didn’t moved for hours (CPU@93%)

Device health checks:
all green but

Some networking issues detected: 
test_upstream_dns: DNS lookup failed for 0.resinio.pool.ntp.org via upstream:
test_upstream_dns: DNS lookup failed for api.balena-cloud.com via upstream:

Sometimes also:

test_balena_registry: Could not communicate with registry2.balena-cloud.com for authentication


root@f1e23ab:~# nslookup api.balena-cloud.com
Address 1: f1e23ab

Name:      api.balena-cloud.com
Address 1: ec2-52-73-205-134.compute-1.amazonaws.com
Address 9: 2600:1f18:6600:7f00:28b6:a5e0:57b6:8a01

root@f1e23ab:~# nslookup 0.resinio.pool.ntp.org
Address 1: f1e23ab

Name:      0.resinio.pool.ntp.org
Address 1: time.cloudflare.com
Address 2: time01.nevondo.com
Address 3: nono.com
Address 4: www.kashra.com


root@f1e23ab:~# curl -kv https://registry2.balena-cloud.com
*   Trying
* Connected to registry2.balena-cloud.com ( port 443 (#0)
* found 128 certificates in /etc/ssl/certs/ca-certificates.crt
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* Connection #0 to host registry2.balena-cloud.com left intact

on Firewall i see also a lot of (accepted) outbound TCP/443 and UDP/123 Traffic…

Supervisor State (NOT Target Supervisor State):

  "api_port": 48484,
  "ip_address": "",
  "os_version": "balenaOS 2.83.18+rev1",
  "mac_address": "20:87:...",
  "supervisor_version": "12.11.23", <-- This is target version! According to Dashboard not installed!
  "update_pending": false,
  "update_failed": false,
  "update_downloaded": false,
  "commit": "e2c86f..", <-- This is target version! According to Dashboard not installed!
  "status": "Idle",
  "download_progress": null

Device Diagnostics cancels with error

diagnose failed: exit code undefined

Any idea what to do to revive this zombie? Remark: It is geographically far away, so not easy to reboot and so on :slight_smile:

Hi @ilgra – we can try to restore the Device to your dashboard, if possible. DM m e the UUID of the device when you get a moment. Thanks!