Cannot push to Balena device that appears to be functional: Supervisor API ECONNREFUSED

Product support
,
1

I’m working with a raspberrypi4-64 device that was working fine last week and has had no changes since then. Today I turned it on to fix a bug, but balena is having problems connecting and pushing.

  • The dashboard current version is sitting at “Factory build” and will not change. I have tried pinning, unpinning, etc. and the version will not launch.
  • Local mode fails with the error shown below; it will not connect to the supervisor.

Strangely, some connectivity works, but some does not:

  • :white_check_mark: it shows up in balena scan
  • :white_check_mark: ssh to the supervisor OS
  • :white_check_mark: balena engine shows the supervisor OS container
  • :white_check_mark: verified there are no IP address collisions
  • :white_check_mark: tunneling works (but the service doesn’t work because there’s no containers)
  • :x: Push breaks with the error shown below.

Things that I have tried:

  • Rebooting
  • Purging the device
  • Re-flashing the SD card with the same img as before
  • Re-flashing the SD card with a new image
  • balena push-ing with or without --nocache
  • Creating a separate subnet and added the device to that
  • Using a new SD card
# ✅ latest balena version
» balena --version
12.50.2

# ✅ device appears in balena scan
» sudo balena scan
Scanning for local balenaOS devices... Reporting scan results
-
  host:          <DEVICEID>.local
  address:       10.0.0.17   # ✅ Correct network IP, no collisions
  osVariant:     development
  dockerInfo:
    Containers:        1
    ContainersRunning: 1
    ContainersPaused:  0
    ContainersStopped: 0
    Images:            2
    Driver:            overlay2
    SystemTime:        2021-10-12T17:48:43.158365953Z
    KernelVersion:     5.10.31-v8
    OperatingSystem:   balenaOS 2.85.2+rev3   # ✅ latest raspberrypi4-64 version
    Architecture:      aarch64
  dockerVersion:
    Version:    19.03.29
    ApiVersion: 1.40

# ✅ SSH to supervisor and check container
» balena ssh <DEVICEID>.local
Last login: Tue Oct 12 17:01:02 2021 from 10.0.0.7
root@<DEVICEID>:~# balena ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS                       PORTS               NAMES
d8464d7329b0        balena/aarch64-supervisor:v12.10.10   "/usr/src/app/entry.…"   12 months ago       Up About an hour (healthy)                       balena_supervisor

# ✅ Setting local mode
» balena device local-mode --enable <DEVICEID>
Local mode on device <DEVICEID> is now ENABLED.

# ❌ `balena push`
» balena push <DEVICEID>.local --multi-dockerignore --debug --env ENV1=xxx --env ENV2=xxx --nocache
[Debug]   Using build source directory: .
[Debug]   Pushing to local device: <DEVICEID>.local
[Debug]   Checking we can access device
[Debug]   Sending request to http://10.0.0.17:48484/ping
Retrying "Supervisor API (GET http://10.0.0.17:48484/ping)" after 2.0s (1 of 5) due to: Error: connect ECONNREFUSED 10.0.0.17:48484
Retrying "Supervisor API (GET http://10.0.0.17:48484/ping)" after 4.0s (2 of 5) due to: Error: connect ECONNREFUSED 10.0.0.17:48484
Retrying "Supervisor API (GET http://10.0.0.17:48484/ping)" after 8.0s (3 of 5) due to: Error: connect ECONNREFUSED 10.0.0.17:48484
Retrying "Supervisor API (GET http://10.0.0.17:48484/ping)" after 16s (4 of 5) due to: Error: connect ECONNREFUSED 10.0.0.17:48484
Retrying "Supervisor API (GET http://10.0.0.17:48484/ping)" after 32s (5 of 5) due to: Error: connect ECONNREFUSED 10.0.0.17:48484
Could not communicate with device supervisor at address 10.0.0.17:48484.
Device may not have local mode enabled. Check with:
  balena device local-mode <device-uuid>

ExpectedError: Could not communicate with device supervisor at address 10.0.0.17:48484.
Device may not have local mode enabled. Check with:
  balena device local-mode <device-uuid>
    at Object.deployToDevice (/usr/local/lib/balena-cli/build/utils/device/deploy.js:69:15)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

Any thoughts?

2

Hey @asdf10489, are you still experiencing this issue? If so, it may be worth checking the supervisor logs with journalctl -u balena-supervisor -a to see if there is an issue. This command is available both with local mode enabled and disabled.

3

You could also run diagnostics and healthchecks in the balena Dashboard to see if any issues are highlighted.

6

I got this when some previously pushed containers were causing supervisor to crash. Tried removing the crashing containers and restarting supervisor, but nothing worked. Had to format the sd card of the device :frowning:

8

Let us know when this happens again and follow the steps mentioned by my colleague. I am sure the journalctl logs should have a reason behind why the supervisor is crashing and fixing that should get the device to restore functionality.

10

Same thing here, newly flashed Rpi4 64 bit, development mode (downloaded as development).
trying to push belana sensor. engine is up, but fleet cannot be pushed:

> balena push 192.168.4.123
Retrying "Supervisor API (GET http://192.168.4.123:48484/ping)" after 2.0s (1 of 5) due to: Error: connect ETIMEDOUT 192.168.4.123:48484
Retrying "Supervisor API (GET http://192.168.4.123:48484/ping)" after 4.0s (2 of 5) due to: Error: connect ETIMEDOUT 192.168.4.123:48484
Retrying "Supervisor API (GET http://192.168.4.123:48484/ping)" after 4.0s (3 of 5) due to: Error: connect ETIMEDOUT 192.168.4.123:48484

  host:          844358a.local
  address:       192.168.4.123
  osVariant:     development
  dockerInfo:
    Containers:        1
    ContainersRunning: 1
    ContainersPaused:  0
    ContainersStopped: 0
    Images:            2
    Driver:            overlay2
    SystemTime:        2022-03-02T12:39:30.449846323Z
    KernelVersion:     5.10.83-v8
    OperatingSystem:   balenaOS 2.91.1
    Architecture:      aarch64
  dockerVersion:
    Version:    19.03.30
    ApiVersion: 1.40

Mar 02 12:16:27 844358a balena-supervisor[2514]: [api]     GET /v1/healthy 200 - 3.898 ms
Mar 02 12:21:27 844358a balena-supervisor[2514]: [api]     GET /v1/healthy 200 - 5.320 ms
Mar 02 12:25:50 844358a balena-supervisor[2514]: [debug]   Attempting container log timestamp flush...
Mar 02 12:25:50 844358a balena-supervisor[2514]: [debug]   Container log timestamp flush complete
Mar 02 12:26:28 844358a balena-supervisor[2514]: [api]     GET /v1/healthy 200 - 3.862 ms
Mar 02 12:31:29 844358a balena-supervisor[2514]: [api]     GET /v1/healthy 200 - 3.054 ms
Mar 02 12:35:50 844358a balena-supervisor[2514]: [debug]   Attempting container log timestamp flush...
Mar 02 12:35:50 844358a balena-supervisor[2514]: [debug]   Container log timestamp flush complete
Mar 02 12:36:30 844358a balena-supervisor[2514]: [api]     GET /v1/healthy 200 - 4.241 ms
Mar 02 12:41:30 844358a balena-supervisor[2514]: [api]     GET /v1/healthy 200 - 3.854 ms
Mar 02 12:45:50 844358a balena-supervisor[2514]: [debug]   Attempting container log timestamp flush...
Mar 02 12:45:50 844358a balena-supervisor[2514]: [debug]   Container log timestamp flush complete
Mar 02 12:46:31 844358a balena-supervisor[2514]: [api]     GET /v1/healthy 200 - 4.067 ms
root@844358a:~#

root@844358a:~# balena ps
CONTAINER ID        IMAGE                                                            COMMAND                  CREATED             STATUS                 PORTS               NAMES
2d64b793ea2f        registry2.balena-cloud.com/v2/f90c95c24a5b648aff6a72250f834c27   "/usr/src/app/entry.…"   5 hours ago         Up 5 hours (healthy)                       balena_supervisor
root@844358a:~#

balena-cli version
13.1.11
11

Upgrade to balena-cli 13.3.0 - did not help, but I guess I found the error:
Trying to access http://192.168.4.123:48484/ping does not work on the network, but a “locale” ping on the device

root@844358a:~# curl 127.0.0.1:48484/ping
OKroot@844358a:~#

works.
The management port 48484 seems not be exposed on the development device - hence balenaCli cannot reach it.

Could be the error?
@klutchell
@vipulgupta2048

12

Hm, I also tried this now with a production image… but it did also not work.
I now try a last time with a development image bound to my balena account.
If this does not work I will try to see whether I can download an old Rpi4 image, because it seems like these cannot be deployed too for me…

13

@nmaas87 my experience has been that this is caused by unreported outages on Balena’s side, which seems to happen quite often. You might wait a bit and try again.

14

Hm that would be troubling.
In my first try I was trying to deploy locally without using the belana services, so that should work without any balena server ^^’

15

Ok, when I deploy a device with production image or development image via balena cloud I can
a) build/push with the prod image via the fleet name (via VPN)
b) build/push with the dev image via the fleet name (via VPN)
c) build/push with the dev image via the locale ip IF it has been set beforehand to locale mode. In this config, also the locale get “/ping” works perfectly.

it seems like the “dev image” downloaded from the website is not yet properly set it into locale mode - the management port is not reachable and hence it cannot be used at all with balena cli.

16

Hey @nmaas87, it is expected that development and production images are not in local mode by default. For all images local mode needs to be enabled manually for local development:

You can also see the differences between production and development images here:

Let us know if there is anything else we can do to assist!

18

Hey there, thanks for the updates.
I guess there is still something missing - or I am not seeing it.
My normal usecase is as follows: I download the development image from the balena.io/os webpage and want to push an idea via balena-cli, completely bypassing the balenaCloud or openBalena.
How do I enable this locale instance without any balena/online capability to allow for a locale push via balena cli?

Thanks!

19

If you ssh into your device you can set localMode: true in your /mnt/boot/config.json file to enable local mode for unmanaged images. This will require a reboot to take effect.

jq '.localMode = true' /mnt/boot/config.json > /mnt/data/config.json && mv /mnt/data/config.json /mnt/boot/config.json

It does seem odd that this setting is not available via the CLI. I’ll make a note of it so we can see if others are encountering similar friction.

21

Thanks, I think I need to set this value before the first boot.
Otherwise the supervisor will find out I modified the config.txt, automatically restore it from the sqlite and reboot again, locking me out before I can ever deploy something.
At least I think it will go like this, because I had a similar issue before which could only be resolved by actually hacking away in the sqlite database and overwriting parameters there.

In the past the locale mode was always enabled for development images, as they were designed to be developed with - and even without balenaCloud (there are a lot of usecases where you must not upload any code to the internet, but could build/deploy the image locally on dev or prod images and would like to use balenaEngine instead of Docker - but making that to hard could frustrate people and make them give up on balena :frowning: ).

Maybe you should put a warning on balenaOS - Run Docker containers on embedded IoT devices that you need to modify the images before you can even use them - just to help people :slight_smile:

22

Hey @nmaas87, it seems I mislead you with my last comment. You should not need to manually enable local mode if you downloaded a development image from balena.io/os.

The supervisor should detect that your image is unmanaged and enable local mode by default:

Can you confirm that your device is in development mode? A good way to test is see if the docker socket is listening for external connections:

DOCKER_HOST=<dut-ip>:2375 docker info

Also you would see developmentMode: true in your config.json.

1 Like
23

Sorry, I will need to re-flash and try again tomorrow, I will get back at you :slight_smile:

25

Sorry for the wait, I tried again.
As yesterday, I used the latest rpi4 64 bit image available on the balena.io/os page in the development version:

ID="balena-os"
NAME="balenaOS"
VERSION="2.91.1"
VERSION_ID="2.91.1"
PRETTY_NAME="balenaOS 2.91.1"
MACHINE="raspberrypi4-64"
META_BALENA_VERSION="2.91.1"

Before booting the image, config.json had following entries:

{
  "deviceType": "raspberrypi4-64",
  "developmentMode": true
}

After the boot, i could find it with the balena-cli

-
  host:          ee5a859.local
  address:       192.168.4.123
  osVariant:     development
  dockerInfo:
    Containers:        1
    ContainersRunning: 1
    ContainersPaused:  0
    ContainersStopped: 0
    Images:            2
    Driver:            overlay2
    SystemTime:        2022-03-03T07:52:45.860152066Z
    KernelVersion:     5.10.83-v8
    OperatingSystem:   balenaOS 2.91.1
    Architecture:      aarch64
  dockerVersion:
    Version:    19.03.30
    ApiVersion: 1.40

Pushing did NOT work

balena push 192.168.4.123
Retrying "Supervisor API (GET http://192.168.4.123:48484/ping)" after 2.0s (1 of 5) due to: Error: connect ETIMEDOUT 192.168.4.123:48484

Trying to access http://192.168.4.123:48484/ping does not work, as yesterday.

A local curl on the device itself for both network and loopback address worked:

root@ee5a859:/resin-boot# curl http://192.168.4.123:48484/ping
OK
root@ee5a859:/resin-boot# curl http://127.0.0.1:48484/ping
OKroot@ee5a859:/resin-boot#

The config.json seems to be correct:

{
  "deviceType": "raspberrypi4-64",
  "localMode": true,
  "persistentLogging": false
}

however…

root@ee5a859:/resin-boot# balena ps
CONTAINER ID        IMAGE                                                            COMMAND                  CREATED             STATUS                    PORTS               NAMES
03e37577d3fc        registry2.balena-cloud.com/v2/f90c95c24a5b648aff6a72250f834c27   "/usr/src/app/entry.…"   15 minutes ago      Up 15 minutes (healthy)                       balena_supervisor

… the management port is not reachable from the outside / balena-cli cannot connect:

root@ee5a859:/resin-boot# balena inspect 03e37577d3fc
[
    {
        "Id": "03e37577d3fc50be57f1a4f298993d45181eed43ab482aa915582ff22b06d693",
        "Created": "2022-03-03T07:43:43.167475445Z",
        "Path": "/usr/src/app/entry.sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2549,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2022-03-03T07:43:45.603555665Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "Health": {
                "Status": "healthy",
                "FailingStreak": 0,
                "Log": [
                    {
                        "Start": "2022-03-03T07:48:45.604397727Z",
                        "End": "2022-03-03T07:48:46.274456357Z",
                        "ExitCode": 0,
                        "Output": "OK"
                    },
                    {
                        "Start": "2022-03-03T07:53:46.317852344Z",
                        "End": "2022-03-03T07:53:46.945851011Z",
                        "ExitCode": 0,
                        "Output": "OK"
                    },
                    {
                        "Start": "2022-03-03T07:58:46.988643573Z",
                        "End": "2022-03-03T07:58:47.636447091Z",
                        "ExitCode": 0,
                        "Output": "OK"
                    }
                ]
            }
        },
        "Image": "sha256:fc5ac88b8f3dc3caeb48a27da2f01ff8f0a2f12e46a886fa87f1e0b7ce191e21",
        "ResolvConfPath": "/var/lib/docker/containers/03e37577d3fc50be57f1a4f298993d45181eed43ab482aa915582ff22b06d693/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/03e37577d3fc50be57f1a4f298993d45181eed43ab482aa915582ff22b06d693/hostname",
        "HostsPath": "/var/lib/docker/containers/03e37577d3fc50be57f1a4f298993d45181eed43ab482aa915582ff22b06d693/hosts",
        "LogPath": "",
        "Name": "/balena_supervisor",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "ContainerIDEnv": "SUPERVISOR_CONTAINER_ID",
            "LogConfig": {
                "Type": "journald",
                "Config": {}
            },
            "NetworkMode": "host",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "/var/run/balena-engine.sock",
                    "Target": "/var/run/balena-engine.sock"
                },
                {
                    "Type": "bind",
                    "Source": "/mnt/boot/config.json",
                    "Target": "/boot/config.json"
                },
                {
                    "Type": "bind",
                    "Source": "/mnt/data/apps.json",
                    "Target": "/boot/apps.json"
                },
                {
                    "Type": "bind",
                    "Source": "/resin-data/balena-supervisor",
                    "Target": "/data"
                },
                {
                    "Type": "bind",
                    "Source": "/proc/net/fib_trie",
                    "Target": "/mnt/fib_trie"
                },
                {
                    "Type": "bind",
                    "Source": "/var/log/supervisor-log",
                    "Target": "/var/log"
                },
                {
                    "Type": "bind",
                    "Source": "/etc/ssl/certs",
                    "Target": "/etc/ssl/certs",
                    "ReadOnly": true
                },
                {
                    "Type": "bind",
                    "Source": "/usr/share/ca-certificates",
                    "Target": "/usr/share/ca-certificates",
                    "ReadOnly": true
                },
                {
                    "Type": "bind",
                    "Source": "/",
                    "Target": "/mnt/root"
                }
            ],
            "MaskedPaths": null,
            "ReadonlyPaths": null
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/0a485cb79959c67f8d065a4d5b50a0147d3d705aa9da27054555d75c632b52dc-init/diff:/var/lib/docker/overlay2/6a12cfa7b83ffcee54650a8cc38d1566f122d1e080e114f46d1639c58a28e92e/diff:/var/lib/docker/overlay2/0a0543afa7ab54aba3be49bb5e5532ce675ed38a00a1a9ed2aca36e340446504/diff:/var/lib/docker/overlay2/733a42cf7136ace0ef0001b1c0c99904036dfc1d362ca0493e548e265699fd75/diff:/var/lib/docker/overlay2/20f3067409f0aed9c09c7f95b74876221999cf1859e442641888f9fab9f15dfc/diff:/var/lib/docker/overlay2/e62ba7c9cac868a66cd05b9900e382142fbf9cc946fad65cd7c6afca1d5ef79b/diff:/var/lib/docker/overlay2/a1eb4524e34fa65c477f2923b8ed9ab0a25976c7bf7d7986a4147155c61364da/diff:/var/lib/docker/overlay2/53d029b0349c0c1a54614183b2a3cb26fe0a7848efccd715750a14ebf9bd63ca/diff:/var/lib/docker/overlay2/a5c54b9757c25e7bf685785940a8b1760ba3b407776aa00369587ced041cf6fb/diff:/var/lib/docker/overlay2/84d1df6d5ff72afaa4aea33944f13528e6b428f4993adf50473963413c95efb9/diff:/var/lib/docker/overlay2/36138e93d2e7e56fe271e4caa54264166e27c85f38f6dc8ac20aaab9daec5573/diff:/var/lib/docker/overlay2/649e19bc2cd34a67ba021be9796fb317145b4f024d0da295d452958617ea775f/diff:/var/lib/docker/overlay2/ce89c6f3a340dbbbf0610b5927ace165b7f08e9364a9ddd6b7700de6b465128c/diff:/var/lib/docker/overlay2/137053d513737aca9bbfa5bd6ce90101427058993c54447a42a38b1aa8f96d0a/diff",
                "MergedDir": "/var/lib/docker/overlay2/0a485cb79959c67f8d065a4d5b50a0147d3d705aa9da27054555d75c632b52dc/merged",
                "UpperDir": "/var/lib/docker/overlay2/0a485cb79959c67f8d065a4d5b50a0147d3d705aa9da27054555d75c632b52dc/diff",
                "WorkDir": "/var/lib/docker/overlay2/0a485cb79959c67f8d065a4d5b50a0147d3d705aa9da27054555d75c632b52dc/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/mnt/boot/config.json",
                "Destination": "/boot/config.json",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/log/supervisor-log",
                "Destination": "/var/log",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/ssl/certs",
                "Destination": "/etc/ssl/certs",
                "Mode": "",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/run/balena-engine.sock",
                "Destination": "/var/run/balena-engine.sock",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/resin-data/balena-supervisor",
                "Destination": "/data",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/proc/net/fib_trie",
                "Destination": "/mnt/fib_trie",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/usr/share/ca-certificates",
                "Destination": "/usr/share/ca-certificates",
                "Mode": "",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/",
                "Destination": "/mnt/root",
                "Mode": "",
                "RW": true,
                "Propagation": "rslave"
            },
            {
                "Type": "bind",
                "Source": "/mnt/data/apps.json",
                "Destination": "/boot/apps.json",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "ee5a859",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "BOOT_MOUNTPOINT=/mnt/boot",
                "DELTA_ENDPOINT=",
                "LED_FILE=/sys/class/leds/led0/brightness",
                "NODE_EXTRA_CA_CERTS=",
                "DOCKER_ROOT=/mnt/root/var/lib/docker",
                "DOCKER_SOCKET=/var/run/balena-engine.sock",
                "MIXPANEL_TOKEN=",
                "LISTEN_PORT=",
                "SUPERVISOR_IMAGE=registry2.balena-cloud.com/v2/f90c95c24a5b648aff6a72250f834c27",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "CONFIG_MOUNT_POINT=/boot/config.json",
                "VERSION=master",
                "DEFAULT_MIXPANEL_TOKEN=bananasbananas",
                "SUPERVISOR_CONTAINER_ID=03e37577d3fc50be57f1a4f298993d45181eed43ab482aa915582ff22b06d693"
            ],
            "Cmd": [
                "/usr/src/app/entry.sh"
            ],
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "wget http://127.0.0.1:${LISTEN_PORT:-48484}/v1/healthy -O - -q"
                ],
                "Interval": 300000000000,
                "Timeout": 30000000000,
                "StartPeriod": 60000000000,
                "Retries": 3
            },
            "Image": "registry2.balena-cloud.com/v2/f90c95c24a5b648aff6a72250f834c27",
            "Volumes": {
                "/data": {}
            },
            "WorkingDir": "/usr/src/app",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "io.balena.qemu.version": "4.0.0+balena-aarch64"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "75ce2abb636ab5db26273f368cd12226ec575cd6e7278423cb4dde7bdd07bede",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/balena-engine/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "527d8e9ba006c376ac53d7996cbad172ed3eb9d9c71fe8a8b7baa806ccaf3a7f",
                    "EndpointID": "c336325d44febbfc0c2cff45757299afc0dede55f43b236b77239f2a64fd91f9",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

So basically, yes, the rpi4-64bit dev image does correctly set itself to localeMode - but the supervisor port does not react to any commands from the outside, which results in it failing when trying to push an image via balena-cli.

I rebooted the Rpi - and AFTER the reboot, this fixed itself:

root@ee5a859:~# balena ps
CONTAINER ID        IMAGE                                                            COMMAND                  CREATED             STATUS                                 PORTS               NAMES
03e37577d3fc        registry2.balena-cloud.com/v2/f90c95c24a5b648aff6a72250f834c27   "/usr/src/app/entry.…"   20 minutes ago      Up About a minute (health: starting)                       balena_supervisor
root@ee5a859:~# balena inspect 03e37577d3fc
[
    {
        "Id": "03e37577d3fc50be57f1a4f298993d45181eed43ab482aa915582ff22b06d693",
        "Created": "2022-03-03T07:43:43.167475445Z",
        "Path": "/usr/src/app/entry.sh",
        "Args": [],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2107,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2022-03-03T08:02:56.057103139Z",
            "FinishedAt": "2022-03-03T08:02:18.052582373Z",
            "Health": {
                "Status": "starting",
                "FailingStreak": 0,
                "Log": [
                    {
                        "Start": "2022-03-03T07:48:45.604397727Z",
                        "End": "2022-03-03T07:48:46.274456357Z",
                        "ExitCode": 0,
                        "Output": "OK"
                    },
                    {
                        "Start": "2022-03-03T07:53:46.317852344Z",
                        "End": "2022-03-03T07:53:46.945851011Z",
                        "ExitCode": 0,
                        "Output": "OK"
                    },
                    {
                        "Start": "2022-03-03T07:58:46.988643573Z",
                        "End": "2022-03-03T07:58:47.636447091Z",
                        "ExitCode": 0,
                        "Output": "OK"
                    }
                ]
            }
        },
        "Image": "sha256:fc5ac88b8f3dc3caeb48a27da2f01ff8f0a2f12e46a886fa87f1e0b7ce191e21",
        "ResolvConfPath": "/var/lib/docker/containers/03e37577d3fc50be57f1a4f298993d45181eed43ab482aa915582ff22b06d693/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/03e37577d3fc50be57f1a4f298993d45181eed43ab482aa915582ff22b06d693/hostname",
        "HostsPath": "/var/lib/docker/containers/03e37577d3fc50be57f1a4f298993d45181eed43ab482aa915582ff22b06d693/hosts",
        "LogPath": "",
        "Name": "/balena_supervisor",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "ContainerIDEnv": "SUPERVISOR_CONTAINER_ID",
            "LogConfig": {
                "Type": "journald",
                "Config": {}
            },
            "NetworkMode": "host",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": true,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "/var/run/balena-engine.sock",
                    "Target": "/var/run/balena-engine.sock"
                },
                {
                    "Type": "bind",
                    "Source": "/mnt/boot/config.json",
                    "Target": "/boot/config.json"
                },
                {
                    "Type": "bind",
                    "Source": "/mnt/data/apps.json",
                    "Target": "/boot/apps.json"
                },
                {
                    "Type": "bind",
                    "Source": "/resin-data/balena-supervisor",
                    "Target": "/data"
                },
                {
                    "Type": "bind",
                    "Source": "/proc/net/fib_trie",
                    "Target": "/mnt/fib_trie"
                },
                {
                    "Type": "bind",
                    "Source": "/var/log/supervisor-log",
                    "Target": "/var/log"
                },
                {
                    "Type": "bind",
                    "Source": "/etc/ssl/certs",
                    "Target": "/etc/ssl/certs",
                    "ReadOnly": true
                },
                {
                    "Type": "bind",
                    "Source": "/usr/share/ca-certificates",
                    "Target": "/usr/share/ca-certificates",
                    "ReadOnly": true
                },
                {
                    "Type": "bind",
                    "Source": "/",
                    "Target": "/mnt/root"
                }
            ],
            "MaskedPaths": null,
            "ReadonlyPaths": null
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/0a485cb79959c67f8d065a4d5b50a0147d3d705aa9da27054555d75c632b52dc-init/diff:/var/lib/docker/overlay2/6a12cfa7b83ffcee54650a8cc38d1566f122d1e080e114f46d1639c58a28e92e/diff:/var/lib/docker/overlay2/0a0543afa7ab54aba3be49bb5e5532ce675ed38a00a1a9ed2aca36e340446504/diff:/var/lib/docker/overlay2/733a42cf7136ace0ef0001b1c0c99904036dfc1d362ca0493e548e265699fd75/diff:/var/lib/docker/overlay2/20f3067409f0aed9c09c7f95b74876221999cf1859e442641888f9fab9f15dfc/diff:/var/lib/docker/overlay2/e62ba7c9cac868a66cd05b9900e382142fbf9cc946fad65cd7c6afca1d5ef79b/diff:/var/lib/docker/overlay2/a1eb4524e34fa65c477f2923b8ed9ab0a25976c7bf7d7986a4147155c61364da/diff:/var/lib/docker/overlay2/53d029b0349c0c1a54614183b2a3cb26fe0a7848efccd715750a14ebf9bd63ca/diff:/var/lib/docker/overlay2/a5c54b9757c25e7bf685785940a8b1760ba3b407776aa00369587ced041cf6fb/diff:/var/lib/docker/overlay2/84d1df6d5ff72afaa4aea33944f13528e6b428f4993adf50473963413c95efb9/diff:/var/lib/docker/overlay2/36138e93d2e7e56fe271e4caa54264166e27c85f38f6dc8ac20aaab9daec5573/diff:/var/lib/docker/overlay2/649e19bc2cd34a67ba021be9796fb317145b4f024d0da295d452958617ea775f/diff:/var/lib/docker/overlay2/ce89c6f3a340dbbbf0610b5927ace165b7f08e9364a9ddd6b7700de6b465128c/diff:/var/lib/docker/overlay2/137053d513737aca9bbfa5bd6ce90101427058993c54447a42a38b1aa8f96d0a/diff",
                "MergedDir": "/var/lib/docker/overlay2/0a485cb79959c67f8d065a4d5b50a0147d3d705aa9da27054555d75c632b52dc/merged",
                "UpperDir": "/var/lib/docker/overlay2/0a485cb79959c67f8d065a4d5b50a0147d3d705aa9da27054555d75c632b52dc/diff",
                "WorkDir": "/var/lib/docker/overlay2/0a485cb79959c67f8d065a4d5b50a0147d3d705aa9da27054555d75c632b52dc/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/mnt/data/apps.json",
                "Destination": "/boot/apps.json",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/mnt/boot/config.json",
                "Destination": "/boot/config.json",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/",
                "Destination": "/mnt/root",
                "Mode": "",
                "RW": true,
                "Propagation": "rslave"
            },
            {
                "Type": "bind",
                "Source": "/var/log/supervisor-log",
                "Destination": "/var/log",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/run/balena-engine.sock",
                "Destination": "/var/run/balena-engine.sock",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/resin-data/balena-supervisor",
                "Destination": "/data",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/etc/ssl/certs",
                "Destination": "/etc/ssl/certs",
                "Mode": "",
                "RW": false,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/proc/net/fib_trie",
                "Destination": "/mnt/fib_trie",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/usr/share/ca-certificates",
                "Destination": "/usr/share/ca-certificates",
                "Mode": "",
                "RW": false,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "ee5a859",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "BOOT_MOUNTPOINT=/mnt/boot",
                "DELTA_ENDPOINT=",
                "LED_FILE=/sys/class/leds/led0/brightness",
                "NODE_EXTRA_CA_CERTS=",
                "DOCKER_ROOT=/mnt/root/var/lib/docker",
                "DOCKER_SOCKET=/var/run/balena-engine.sock",
                "MIXPANEL_TOKEN=",
                "LISTEN_PORT=",
                "SUPERVISOR_IMAGE=registry2.balena-cloud.com/v2/f90c95c24a5b648aff6a72250f834c27",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "CONFIG_MOUNT_POINT=/boot/config.json",
                "VERSION=master",
                "DEFAULT_MIXPANEL_TOKEN=bananasbananas",
                "SUPERVISOR_CONTAINER_ID=03e37577d3fc50be57f1a4f298993d45181eed43ab482aa915582ff22b06d693"
            ],
            "Cmd": [
                "/usr/src/app/entry.sh"
            ],
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "wget http://127.0.0.1:${LISTEN_PORT:-48484}/v1/healthy -O - -q"
                ],
                "Interval": 300000000000,
                "Timeout": 30000000000,
                "StartPeriod": 60000000000,
                "Retries": 3
            },
            "Image": "registry2.balena-cloud.com/v2/f90c95c24a5b648aff6a72250f834c27",
            "Volumes": {
                "/data": {}
            },
            "WorkingDir": "/usr/src/app",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {
                "io.balena.qemu.version": "4.0.0+balena-aarch64"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "e897abde5cc52bc19e43728af2faf119cd164f5e4f62551a3cf0916bb78077b7",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/balena-engine/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "527d8e9ba006c376ac53d7996cbad172ed3eb9d9c71fe8a8b7baa806ccaf3a7f",
                    "EndpointID": "9feef1704be834498394161335db8cf59fed35429f9a6665cd049491c041f768",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

I don’t know what changed, but after a reboot, it works.
I guess there is something needed, like recreating the supervisor or similar after it realized “I am in dev mode and not connected to balena.io, so I need to enable localeMode”. The hard fix would be to add a reboot directly after setting these commands, but you guys probably have a better solution. I would encourage not to leave the image in this state and write “please reboot after first boot” on the box, as it might scare away people :wink:

Hope that analysis helps?

26

Thank you for the detailed analysis @nmaas87! I’m going to attempt to reproduce it on my device, but it seems like the supervisor is not applying the localMode state on first boot.

I’ll respond here when I have narrowed down the issue or have more to share!

1 Like
28

Hey @nmaas87, I’m not able to reproduce this with balenaOS 2.94.4. Can you confirm if it is still an issue with the latest release? I suspect this bug has already been squashed.