Cannot push to Balena device that appears to be functional: Supervisor API ECONNREFUSED

I’m working with a raspberrypi4-64 device that was working fine last week and has had no changes since then. Today I turned it on to fix a bug, but balena is having problems connecting and pushing.

  • The dashboard current version is sitting at “Factory build” and will not change. I have tried pinning, unpinning, etc. and the version will not launch.
  • Local mode fails with the error shown below; it will not connect to the supervisor.

Strangely, some connectivity works, but some does not:

  • :white_check_mark: it shows up in balena scan
  • :white_check_mark: ssh to the supervisor OS
  • :white_check_mark: balena engine shows the supervisor OS container
  • :white_check_mark: verified there are no IP address collisions
  • :white_check_mark: tunneling works (but the service doesn’t work because there’s no containers)
  • :x: Push breaks with the error shown below.

Things that I have tried:

  • Rebooting
  • Purging the device
  • Re-flashing the SD card with the same img as before
  • Re-flashing the SD card with a new image
  • balena push-ing with or without --nocache
  • Creating a separate subnet and added the device to that
  • Using a new SD card
# ✅ latest balena version
» balena --version
12.50.2

# ✅ device appears in balena scan
» sudo balena scan
Scanning for local balenaOS devices... Reporting scan results
-
  host:          <DEVICEID>.local
  address:       10.0.0.17   # ✅ Correct network IP, no collisions
  osVariant:     development
  dockerInfo:
    Containers:        1
    ContainersRunning: 1
    ContainersPaused:  0
    ContainersStopped: 0
    Images:            2
    Driver:            overlay2
    SystemTime:        2021-10-12T17:48:43.158365953Z
    KernelVersion:     5.10.31-v8
    OperatingSystem:   balenaOS 2.85.2+rev3   # ✅ latest raspberrypi4-64 version
    Architecture:      aarch64
  dockerVersion:
    Version:    19.03.29
    ApiVersion: 1.40

# ✅ SSH to supervisor and check container
» balena ssh <DEVICEID>.local
Last login: Tue Oct 12 17:01:02 2021 from 10.0.0.7
root@<DEVICEID>:~# balena ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                       PORTS               NAMES
d8464d7329b0        balena/aarch64-supervisor:v12.10.10   "/usr/src/app/entry.…"   12 months ago       Up About an hour (healthy)                       balena_supervisor

# ✅ Setting local mode
» balena device local-mode --enable <DEVICEID>
Local mode on device <DEVICEID> is now ENABLED.

# ❌ `balena push`
» balena push <DEVICEID>.local --multi-dockerignore --debug --env ENV1=xxx --env ENV2=xxx --nocache
[Debug]   Using build source directory: .
[Debug]   Pushing to local device: <DEVICEID>.local
[Debug]   Checking we can access device
[Debug]   Sending request to http://10.0.0.17:48484/ping
Retrying "Supervisor API (GET http://10.0.0.17:48484/ping)" after 2.0s (1 of 5) due to: Error: connect ECONNREFUSED 10.0.0.17:48484
Retrying "Supervisor API (GET http://10.0.0.17:48484/ping)" after 4.0s (2 of 5) due to: Error: connect ECONNREFUSED 10.0.0.17:48484
Retrying "Supervisor API (GET http://10.0.0.17:48484/ping)" after 8.0s (3 of 5) due to: Error: connect ECONNREFUSED 10.0.0.17:48484
Retrying "Supervisor API (GET http://10.0.0.17:48484/ping)" after 16s (4 of 5) due to: Error: connect ECONNREFUSED 10.0.0.17:48484
Retrying "Supervisor API (GET http://10.0.0.17:48484/ping)" after 32s (5 of 5) due to: Error: connect ECONNREFUSED 10.0.0.17:48484
Could not communicate with device supervisor at address 10.0.0.17:48484.
Device may not have local mode enabled. Check with:
  balena device local-mode <device-uuid>

ExpectedError: Could not communicate with device supervisor at address 10.0.0.17:48484.
Device may not have local mode enabled. Check with:
  balena device local-mode <device-uuid>
    at Object.deployToDevice (/usr/local/lib/balena-cli/build/utils/device/deploy.js:69:15)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

Any thoughts?

Hey @asdf10489, are you still experiencing this issue? If so, it may be worth checking the supervisor logs with journalctl -u balena-supervisor -a to see if there is an issue. This command is available both with local mode enabled and disabled.

You could also run diagnostics and healthchecks in the balena Dashboard to see if any issues are highlighted.