Supervisor Connection Refused

Product support balenaOS
,
1

I updated a Raspberry Pi 3 B v1.2 with the newest available OS and now I can’t access the supervisor API from a container on the device. It was working fine before the update.

OS: balenaOS 2.52.7+rev1
Supervisor: 11.9.4

Also, I can access the API remotely, but not from the container. The device is running the same application as before the update, nothing else changed.

the docker-compose.yml file includes the proper label to access the supervisor api:

labels:
  io.balena.features.supervisor-api: '1'

I granted support access on the device, feel free to access it.

3

Hi @nick.slocum thats weird, can you tell me which endpoint you are testing on. I just updated my pi4 to the same version and running the v1/device endpoint seems to work:

root@c182f89:/# env | grep BALENA_HOST
BALENA_HOST_OS_VERSION=balenaOS 2.52.7+rev1
root@c182f89:/# curl -X GET --header "Content-Type:application/json" \
>     "$BALENA_SUPERVISOR_ADDRESS/v1/device?apikey=$BALENA_SUPERVISOR_API_KEY"
{"api_port":48484,"ip_address":"192.168.1.83 192.168.1.47","os_version":"balenaOS 2.52.7+rev1","mac_address":"DC:A6:32:06:58:B0 DC:A6:32:06:58:B1","supervisor_version":"11.9.4","update_pending":false,"update_failed":false,"update_downloaded":false,"commit":"44f214fa13cd46ad78d0f22217a4464a","status":"Idle","download_progress":null}

Could you try running the same simple curl command, just the below:

curl -X GET --header "Content-Type:application/json" \
    "$BALENA_SUPERVISOR_ADDRESS/v1/device?apikey=$BALENA_SUPERVISOR_API_KEY"
4

Also looking at the change log for the latest supervisor version ( https://github.com/balena-io/balena-supervisor/blob/master/CHANGELOG.md#2020-07-20 ) it looks like some fixes were applied to fix purge and restart so if you are using either of those two endpoints it could be that you hit a bug in supervisor 11.9.4

7

I tested a couple API calls and they all return the same error:

curl: (7) Failed to connect to 127.0.0.1 port 48484: Connection refused

Here’s the call you requested:

bash-5.0# curl -X GET --header "Content-Type:application/json" \
>     "$BALENA_SUPERVISOR_ADDRESS/v1/device?apikey=$BALENA_SUPERVISOR_API_KEY"
curl: (7) Failed to connect to 127.0.0.1 port 48484: Connection refused
8

to be clear, I made all the API calls from the service that has the proper label defined:

services:
  api:
    build: ./api
    labels:
      io.balena.features.supervisor-api: '1'
      io.balena.features.sysfs: '1'
    restart: always
9

That looks like the supervisor is not running correctly, can you run the “diagnostics checks” on the device. You can find these on the left hand side menu when viewing the device

10

Sure thing. I just ran the health checks and everything came back Succeeeded/green

11

Okay, that is very interesting. If you can send me the link to the device in a private message I can have a look. It sounds like perhaps somehow the supervisor is not connected to the docker bridge or perhaps the supervisor api key is not being injected into the environment correctly. I will see what I can find out

15

@@nick.slocum after looking at your device I haven’t yet found anything different in your device versus my device running the same OS version. I would like to try reproduce this by upgrading to this version, can you remember what OS version you were running on the device before upgrading 2.52.7?

17

Before upgrading, the device was running:

OS: balenaOS 2.51.1+rev1
Supervisor: 11.4.10

Those versions were the most recent for this device, prior to 2.52.7 being released.

20

Hi @nick.slocum I managed to reproduce this, it seems that either due to the balena-engine upgrade or some supervisor changes that in the 2.52.7 version of the OS, the supervisor API is only reachable if you specify network_mode: host to the host that requires access to the API. So as a work around now you could add that, but we will be removing the 2.52.7 version of the OS from production as this is a critical bug that we will need to fix and also further improve our testing to cover this case.

21

Thanks for the quick analysis.
For security reasons, I can’t expose that service to the host network. Fortunately, the previous release works just fine for us, so I’ll continue to use that for our fleet.

thanks again!

22

No problem at all, we will be pulling that release and for now I recommend sticking on the older release, we will hopefully release a fixed version in the coming week or two and then will inform you here. Sorry for the inconvenience