supervisor 19.0.8 upgrade issues

Product support balenaOS
1

Hello all,

I have just tried upgrading the supervisor on some of my devices to Supervisor v19.0.8 ..

There were no errors I saw during the process, but the devices lost heartbeat while maintaining cloudlink.

Looking at the supervisor logs, it appears there is a problem submitting status:
Aug 22 10:26:16 3cbbdcd balena-supervisor[19466]: [debug] Handling of local mode switch is completed
Aug 22 10:26:16 3cbbdcd balena-supervisor[19466]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Aug 22 10:26:16 3cbbdcd balena-supervisor[19466]: [info] API Binder bound to: http://127.0.0.1:48484/v7/
Aug 22 10:26:16 3cbbdcd balena-supervisor[19466]: [event] Event: Supervisor start {}
Aug 22 10:26:16 3cbbdcd balena-supervisor[19466]: [info] Starting API server
Aug 22 10:26:16 3cbbdcd balena-supervisor[19466]: [debug] Ensuring device is provisioned
Aug 22 10:26:16 3cbbdcd balena-supervisor[19466]: [info] Supervisor API successfully started on port 48480
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [debug] Connectivity check enabled: true
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [debug] VPN status path exists.
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [info] VPN connection is active.
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [info] Internet Connectivity: OK
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [debug] Starting periodic check for IP addresses
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [info] Reporting initial state, supervisor version and API info
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [debug] Skipping preloading
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [info] Applying target state
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [debug] Starting current state report
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [debug] Starting target state poll
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [debug] Spawning journalctl -a --follow -S 2026-08-22 10:26:11 -o json _SYSTEMD_UNIT=balena.service _SYSTEMD_UNIT=os-power-mode.service _SYSTEMD_UNIT=os-fan-profile.service _SYSTEMD_UNIT=nvpmodel.service _SYSTEMD_UNIT=jetson-qspi-manager.service _SYSTEMD_UNIT=os-update.service
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [debug] Could not retrieve last reported state from cache, proceeding with empty cache
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [error] Target state poll failed HTTPError: Request failed with status code 503 (Service Unavailable): GET http://127.0.0.1:48484/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [error] at Request. (/usr/src/app/dist/583.app.js:2:206118)
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [error] at Object.onceWrapper (node:events:634:26)
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [error] at Request.emit (node:events:531:35)
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [error] at Request._onResponseBase (/usr/src/app/dist/583.app.js:2:190829)
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [error] at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [error] at async Request._onResponse (/usr/src/app/dist/583.app.js:2:191650)
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [debug] Finished applying target state
Aug 22 10:26:17 3cbbdcd balena-supervisor[19466]: [success] Device state apply success
Aug 22 10:26:32 3cbbdcd balena-supervisor[19466]: [error] Target state poll failed HTTPError: Request failed with status code 503 (Service Unavailable): GET http://127.0.0.1:48484/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state
Aug 22 10:26:32 3cbbdcd balena-supervisor[19466]: [error] at Request. (/usr/src/app/dist/583.app.js:2:206118)
Aug 22 10:26:32 3cbbdcd balena-supervisor[19466]: [error] at Object.onceWrapper (node:events:634:26)
Aug 22 10:26:32 3cbbdcd balena-supervisor[19466]: [error] at Request.emit (node:events:531:35)
Aug 22 10:26:32 3cbbdcd balena-supervisor[19466]: [error] at Request._onResponseBase (/usr/src/app/dist/583.app.js:2:190829)
Aug 22 10:26:32 3cbbdcd balena-supervisor[19466]: [error] at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
Aug 22 10:26:32 3cbbdcd balena-supervisor[19466]: [error] at async Request._onResponse (/usr/src/app/dist/583.app.js:2:191650)
Aug 22 10:26:32 3cbbdcd balena-supervisor[19466]: [error] Device state report failure! Status code: 502 - message: “Target connection failed: error sending request for url (https://api.balena-cloud.com/device/v3/state)”
Aug 22 10:26:32 3cbbdcd balena-supervisor[19466]: [info] Retrying device state report in 15 seconds
Aug 22 10:27:02 3cbbdcd balena-supervisor[19466]: [error] Target state poll failed HTTPError: Request failed with status code 503 (Service Unavailable): GET http://127.0.0.1:48484/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state
Aug 22 10:27:02 3cbbdcd balena-supervisor[19466]: [error] at Request. (/usr/src/app/dist/583.app.js:2:206118)
Aug 22 10:27:02 3cbbdcd balena-supervisor[19466]: [error] at Object.onceWrapper (node:events:634:26)
Aug 22 10:27:02 3cbbdcd balena-supervisor[19466]: [error] at Request.emit (node:events:531:35)
Aug 22 10:27:02 3cbbdcd balena-supervisor[19466]: [error] at Request._onResponseBase (/usr/src/app/dist/583.app.js:2:190829)
Aug 22 10:27:02 3cbbdcd balena-supervisor[19466]: [error] at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
Aug 22 10:27:02 3cbbdcd balena-supervisor[19466]: [error] at async Request._onResponse (/usr/src/app/dist/583.app.js:2:191650)
Aug 22 10:27:03 3cbbdcd balena-supervisor[19466]: [error] Device state report failure! Status code: 502 - message: “Target connection failed: error sending request for url (https://api.balena-cloud.com/device/v3/state)”
Aug 22 10:27:03 3cbbdcd balena-supervisor[19466]: [info] Retrying device state report in 30 seconds
Aug 22 10:27:48 3cbbdcd balena-supervisor[19466]: [error] Device state report failure! Status code: 502 - message: “Target connection failed: error sending request for url (https://api.balena-cloud.com/device/v3/state)”
Aug 22 10:27:48 3cbbdcd balena-supervisor[19466]: [info] Retrying device state report in 60 seconds
Aug 22 10:28:02 3cbbdcd balena-supervisor[19466]: [error] Target state poll failed HTTPError: Request failed with status code 503 (Service Unavailable): GET http://127.0.0.1:48484/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state
Aug 22 10:28:02 3cbbdcd balena-supervisor[19466]: [error] at Request. (/usr/src/app/dist/583.app.js:2:206118)
Aug 22 10:28:02 3cbbdcd balena-supervisor[19466]: [error] at Object.onceWrapper (node:events:634:26)
Aug 22 10:28:02 3cbbdcd balena-supervisor[19466]: [error] at Request.emit (node:events:531:35)
Aug 22 10:28:02 3cbbdcd balena-supervisor[19466]: [error] at Request._onResponseBase (/usr/src/app/dist/583.app.js:2:190829)
Aug 22 10:28:02 3cbbdcd balena-supervisor[19466]: [error] at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
Aug 22 10:28:02 3cbbdcd balena-supervisor[19466]: [error] at async Request._onResponse (/usr/src/app/dist/583.app.js:2:191650)
Aug 22 10:29:03 3cbbdcd balena-supervisor[19466]: [error] Device state report failure! Status code: 502 - message: “Target connection failed: error sending request for url (https://api.balena-cloud.com/device/v3/state)”
Aug 22 10:29:03 3cbbdcd balena-supervisor[19466]: [info] Retrying device state report in 120 seconds
Aug 22 10:30:02 3cbbdcd balena-supervisor[19466]: [error] Target state poll failed HTTPError: Request failed with status code 503 (Service Unavailable): GET http://127.0.0.1:48484/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state
Aug 22 10:30:02 3cbbdcd balena-supervisor[19466]: [error] at Request. (/usr/src/app/dist/583.app.js:2:206118)
Aug 22 10:30:02 3cbbdcd balena-supervisor[19466]: [error] at Object.onceWrapper (node:events:634:26)
Aug 22 10:30:02 3cbbdcd balena-supervisor[19466]: [error] at Request.emit (node:events:531:35)
Aug 22 10:30:02 3cbbdcd balena-supervisor[19466]: [error] at Request._onResponseBase (/usr/src/app/dist/583.app.js:2:190829)
Aug 22 10:30:02 3cbbdcd balena-supervisor[19466]: [error] at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
Aug 22 10:30:02 3cbbdcd balena-supervisor[19466]: [error] at async Request._onResponse (/usr/src/app/dist/583.app.js:2:191650)
Aug 22 10:31:18 3cbbdcd balena-supervisor[19466]: [error] Device state report failure! Status code: 502 - message: “Target connection failed: error sending request for url (https://api.balena-cloud.com/device/v3/state)”
Aug 22 10:31:18 3cbbdcd balena-supervisor[19466]: [info] Retrying device state report in 240 seconds
Aug 22 10:34:02 3cbbdcd balena-supervisor[19466]: [error] Target state poll failed HTTPError: Request failed with status code 503 (Service Unavailable): GET http://127.0.0.1:48484/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state
Aug 22 10:34:02 3cbbdcd balena-supervisor[19466]: [error] at Request. (/usr/src/app/dist/583.app.js:2:206118)
Aug 22 10:34:02 3cbbdcd balena-supervisor[19466]: [error] at Object.onceWrapper (node:events:634:26)
Aug 22 10:34:02 3cbbdcd balena-supervisor[19466]: [error] at Request.emit (node:events:531:35)
Aug 22 10:34:02 3cbbdcd balena-supervisor[19466]: [error] at Request._onResponseBase (/usr/src/app/dist/583.app.js:2:190829)
Aug 22 10:34:02 3cbbdcd balena-supervisor[19466]: [error] at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
Aug 22 10:34:02 3cbbdcd balena-supervisor[19466]: [error] at async Request._onResponse (/usr/src/app/dist/583.app.js:2:191650)
Aug 22 10:35:34 3cbbdcd balena-supervisor[19466]: [error] Device state report failure! Status code: 502 - message: “Target connection failed: error sending request for url (https://api.balena-cloud.com/device/v3/state)”
Aug 22 10:35:34 3cbbdcd balena-supervisor[19466]: [info] Retrying device state report in 480 seconds

logs of the new balena-next service are similar:
INFO api:request: close time.busy=168µs time.idle=33.5µs method=GET uri=/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state version=HTTP/1.1 status=503 Service Unavailable
WARN api:request: response failed classification=Status code: 502 Bad Gateway latency=15017 ms method=PATCH uri=/device/v3/state version=HTTP/1.1 status=502 Bad Gateway
INFO api:request: close time.busy=1.01ms time.idle=15.0s method=PATCH uri=/device/v3/state version=HTTP/1.1 status=502 Bad Gateway
WARN api:request: response failed classification=Status code: 503 Service Unavailable latency=0 ms method=GET uri=/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state version=HTTP/1.1 status=503 Service Unavailable
INFO api:request: close time.busy=135µs time.idle=43.1µs method=GET uri=/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state version=HTTP/1.1 status=503 Service Unavailable
WARN api:request: response failed classification=Status code: 502 Bad Gateway latency=15017 ms method=PATCH uri=/device/v3/state version=HTTP/1.1 status=502 Bad Gateway
INFO api:request: close time.busy=1.01ms time.idle=15.0s method=PATCH uri=/device/v3/state version=HTTP/1.1 status=502 Bad Gateway
WARN api:request: response failed classification=Status code: 502 Bad Gateway latency=15016 ms method=PATCH uri=/device/v3/state version=HTTP/1.1 status=502 Bad Gateway
INFO api:request: close time.busy=993µs time.idle=15.0s method=PATCH uri=/device/v3/state version=HTTP/1.1 status=502 Bad Gateway
INFO api:request: close time.busy=78.0µs time.idle=61.5µs method=GET uri=/ping version=HTTP/1.1 status=200 OK
WARN api:request: response failed classification=Status code: 503 Service Unavailable latency=0 ms method=GET uri=/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state version=HTTP/1.1 status=503 Service Unavailable
INFO api:request: close time.busy=185µs time.idle=38.7µs method=GET uri=/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state version=HTTP/1.1 status=503 Service Unavailable
WARN poll:get: error=request failed with: error sending request for url (https://api.balena-cloud.com/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state): hyper_util::client::legacy::Error(Connect, ConnectError(“dns error”, Custom { kind: Uncategorized, error: “failed to lookup address information: Try again” })) … will retry in 640s
WARN api:request: response failed classification=Status code: 502 Bad Gateway latency=15017 ms method=PATCH uri=/device/v3/state version=HTTP/1.1 status=502 Bad Gateway
INFO api:request: close time.busy=1.15ms time.idle=15.0s method=PATCH uri=/device/v3/state version=HTTP/1.1 status=502 Bad Gateway
WARN api:request: response failed classification=Status code: 503 Service Unavailable latency=0 ms method=GET uri=/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state version=HTTP/1.1 status=503 Service Unavailable
INFO api:request: close time.busy=179µs time.idle=41.6µs method=GET uri=/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state version=HTTP/1.1 status=503 Service Unavailable
INFO api:request: close time.busy=29.6µs time.idle=37.0µs method=GET uri=/ping version=HTTP/1.1 status=200 OK
WARN api:request: response failed classification=Status code: 502 Bad Gateway latency=15016 ms method=PATCH uri=/device/v3/state version=HTTP/1.1 status=502 Bad Gateway
INFO api:request: close time.busy=967µs time.idle=15.0s method=PATCH uri=/device/v3/state version=HTTP/1.1 status=502 Bad Gateway
INFO api:request: close time.busy=566µs time.idle=29.7s method=GET uri=/v1/healthy version=HTTP/1.1
ERROR poll: poll failed: cancelled
DEBUG poll:get: close time.busy=8.44ms time.idle=934s cancelled=true
INFO api:request: close time.busy=32.1µs time.idle=39.7µs method=GET uri=/ping version=HTTP/1.1 status=200 OK
WARN api:request: response failed classification=Status code: 503 Service Unavailable latency=0 ms method=GET uri=/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state version=HTTP/1.1 status=503 Service Unavailable
INFO api:request: close time.busy=146µs time.idle=34.7µs method=GET uri=/device/v3/3cbbdcd6a68e0269247945d2c6737ddb/state version=HTTP/1.1 status=503 Service Unavailable
INFO api:request: close time.busy=34.9µs time.idle=65.0µs method=GET uri=/ping version=HTTP/1.1 status=200 OK
WARN api:request: response failed classification=Status code: 502 Bad Gateway latency=15017 ms method=PATCH uri=/device/v3/state version=HTTP/1.1 status=502 Bad Gateway
INFO api:request: close time.busy=1.08ms time.idle=15.0s method=PATCH uri=/device/v3/state version=HTTP/1.1 status=502 Bad Gateway

I can curl all the public addresses, ping, etc - i dont think it is name resolution ..

Thoughts?

2

I think I have isolated it to a problem in the bridge network for the new supervisor containers ..

i can ping the Balena VPN interface and the local NIC IP, but not the local router from inside any of the new containers.

i can from the host OS ..

3

I have done a lot more investigating ..

Ping packets to the local LAN router are being consumed or dropped between the bridge and the host’s routing/forwarding path — before iptables, before the physical interface.

The fleet with these issues includes a device that was recently installed and has the latest BalenaOS on it (balenaOS 7.4.0+rev7) as well as some with older ones ..

4

FIXED!

I had configured firewall rules in a container using nf_tables .. but BalenaOS uses iptables (legacy) ..

I ended up moving the affected devices to a different (blank) fleet, stopping the container with the firewall rules, pulling and running a nicolaka/netshoot container and executing a “nft flush ruleset”.

This cleared the offending rules and allowed the device to communicate properly and resume normal operations.

Anyone reading this, obviously this is destructive, I just wanted the devices back and going (had no local data to loose) .. I will now fix up the firewall rules, update that container, deploy and then migrate the devices …