BalenaCloud device resin_supervisor constantly logs 'Error from the API: 401'

Product support
1

Hello,
I have a few devices set up in my account that all run on the same application. I recently set up 3rd-party logging to Papertrail, and through that I can see the resin_supervisor logs.

The supervisor on one of my devices has been throwing a 401 error every 10 seconds for some time now. It didn’t seem to affect functionality at first, but eventually the device failed (or ignored?) a Supervisor update. Since then it has started exhibiting other weird behavior, including not showing one of my application containers in the dashboard “Services” list, even though that container is actively logging in the dashboard log window. The device is also stuck on an older release of my application code.

The error in the logs is the only indication of a possible problem. My application containers are all running and logging fine with no issues (aside from them being an older version than expected). My other two devices are not exhibiting this issue.

The error in the logs is:

f7402c2  [error]   Error from the API: 401
f7402c2  [error]   Non-200 response from the API! Status code: 401 - message: Error
f7402c2  [error]         at /usr/src/app/dist/app.js:22:554770
f7402c2  [error]       at runMicrotasks (<anonymous>)
f7402c2  [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
f7402c2  [error]       at async /usr/src/app/dist/app.js:22:554078
f7402c2  [error]       at async /usr/src/app/dist/app.js:22:555600

I have tried restarting containers, rebooting, and manually pulling the power plug and letting it sit offline for a bit. This is a development/test machine and is not important in the short term, however as I get closer to deployment, I’m worried about this happening in production and what options I would have to solve it in that situation.

Thanks,
Tommy

2

Hey there Tommy, it sounds like something networking-related could be getting in the way of the supervisor reporting its state to the API. The data sent in these updates is ultimately what is presented in the dashboard so it makes sense if these requests are failing that you’re seeing strange or incomplete data in the dashboard.

Could there be any kind of firewall or proxy between your device and balenaCloud that may prevent these requests from reaching our API? You can read more about the network requirements here: Network Setup on balenaOS 2.x - Balena Documentation

Let us know if this helps; if not, it might help if you could enable support access and share the device UUID here so that we can take a look at the device ourselves.

4

Hi @chrisys , thanks for your reply. I don’t have any firewalls or proxies - my network is pretty vanilla outside of PiHole (which I’ve checked, and it isn’t blocking any DNS for the device). I have another device on the local network and it functions fine (granted, that one is in local mode, but Papertrail captures its logs as well). Additionally, I used to run 2 non-local-mode devices and have performed many Supervisor, OS, and app updates without issues before this. The VPN is connected, logs flow to Balena normally, and I can SSH in from the Balena dashboard; all of this leads me to believe it’s likely not a networking problem (plus, the error is 401, which is usually authentication related I think?).

I have granted support access for a week, and the UUID is f7402c253897c86a51dbcae4e617d2ab. Thanks for looking into this!

5

Hey Tommy,

Was the device with issues flashed with the same image as the other 2 devices? If not, is it in the same application?

7

I’ve had 3 devices in the same location:

  1. This problematic device (Pi CM4). It was flashed with development image balena-cloud-[app]-raspberrypicm4-ioboard-2.71.3+rev5-dev-v12.3.5.img.

  2. My main development device (Pi 4, most often in local mode). I believe that device used balena-cloud-[app]-raspberrypi4-64-2.58.6+rev1-dev-v11.14.0.img.

  3. A test device (Pi 3). This ran a development image at first, then later a production image. It is not currently in service.

The images were downloaded at different time (so they were different versions), but all for the same app (and I have not switched apps at all).

All devices have been through multiple successful OS and Supervisor updates.

8

In case it’s useful information, I just set up my test Pi 3 with balena-cloud-[app]-raspberrypi3-64-2.65.0+rev1-dev-v12.2.11.img right next to the CM4 that is throwing errors. It connected to Balena fine, downloaded the latest app release, and started up without issue. I waited a few minutes and then performed a supervisor update to 12.5.10, still all good. It has not logged any API errors thus far.

10

I forgot about this for a bit and let the support access lapse - I’ve re-granted it for a week.

17

Hi @chrisys and @danthegoodman1, I’ve re-enabled support access for another week. Is this something you still want to look into, or should I just re-flash the image and see if it goes away?

19

Hi,
Thanks for re-enabling support access to the device.
Unless this is blocking you, we would like to have some more time investigating the state of this device.

Kind regards,
Thodoris

20

@thgreasi no worries, it’s not blocking me - I’ll keep support access enabled. Thank you!

24

Hey, I am looking into the issue with your device but it is now offline. Are you able to bring it back up and let us know then I’ll resume.

I have observed that the device is healthy and your application is not experiencing any disruption. I didn’t get a chance to confirm if it’s running the target release but I believe it is. This is because I saw the device was successfully asking for updates but it was having an issue reporting the current state. That is why the current commit never changes and other stats like device metrics remained the same.

I won’t get an alert when the device is back online so please provide a good buffer like 1 more week so I can also resume debugging it on Monday if I get access too late into the evening. That would be greatly appreciated!

26

@20k-ultra My apologies, I took it offline briefly to try to track down an unrelated behavior in my system. I have plugged it back in and it is now back online. Thank you!

27

I should have added: please feel free to tinker and change things on the device as you see fit to diagnose. It is a test device and not hooked to anything critical.

Also, I can confirm your statement about receiving the latest release: I log the release level on startup, and at this most recent boot it logged the target release.

28

This is interesting: I just encountered the issue on another one of my devices. It happened after disabling local mode (something I’ve switched back and forth from a few times the past few weeks). I was able to grab the logs from journalctl, but they didn’t look particularly useful to me. Note the 200 responses from /v1/healthy until I disabled local mode at 23:55:43.


May 12 23:42:35 c801556 resin-supervisor[1853]: [debug]   Attempting container log timestamp flush...
May 12 23:42:35 c801556 resin-supervisor[1853]: [debug]   Container log timestamp flush complete
May 12 23:43:04 c801556 resin-supervisor[1853]: [api]     GET /v1/healthy 200 - 58.797 ms
May 12 23:48:06 c801556 resin-supervisor[1853]: [api]     GET /v1/healthy 200 - 7.407 ms
May 12 23:52:35 c801556 resin-supervisor[1853]: [debug]   Attempting container log timestamp flush...
May 12 23:52:35 c801556 resin-supervisor[1853]: [debug]   Container log timestamp flush complete
May 12 23:53:09 c801556 resin-supervisor[1853]: [api]     GET /v1/healthy 200 - 6.007 ms
May 12 23:55:43 c801556 resin-supervisor[1853]: [event]   Event: Update notification {}
May 12 23:55:43 c801556 resin-supervisor[1853]: [api]     POST /v1/update 204 - 48.092 ms
May 12 23:55:44 c801556 resin-supervisor[1853]: [info]    Applying target state
May 12 23:55:44 c801556 resin-supervisor[1853]: [debug]   Cannot parse Volume: f2d02581c2e0fa276c9631292d8de261423d3e23c118b79505f1e360251e2ee2
May 12 23:55:44 c801556 resin-supervisor[1853]: [debug]   Cannot parse Volume: 06fff239e73194c055e9d6c331fe82a03990198354da2fddb575df89f562a5d2
May 12 23:55:44 c801556 resin-supervisor[1853]: [event]   Event: Apply config change in progress {}
May 12 23:55:45 c801556 resin-supervisor[1853]: [info]    Switching logging backend to BalenaLogBackend
May 12 23:55:45 c801556 resin-supervisor[1853]: [event]   Event: Apply config change success {}
May 12 23:55:45 c801556 resin-supervisor[1853]: [warn]    Invalid firewall mode: . Reverting to state: off
May 12 23:55:45 c801556 resin-supervisor[1853]: [info]    🔥 Applying firewall mode: off
May 12 23:55:45 c801556 resin-supervisor[1853]: [debug]   1 containers, 1 images, 0 volumes, 1 networks are linked to current supervisor
May 12 23:55:45 c801556 resin-supervisor[1853]: [debug]   Leaving local mode and cleaning up objects since 2021-05-12T19:28:03.735Z
May 12 23:55:45 c801556 resin-supervisor[1853]: [debug]   Going to delete the following objects: 4 containers, 4 images, 3 volumes, 1 networks
May 12 23:55:46 c801556 resin-supervisor[1853]: [success] 🔥 Firewall mode applied
May 12 23:55:50 c801556 resin-supervisor[1853]: [event]   Event: Service exit {"service":{"appId":1,"serviceId":3,"serviceName":"wifi-connect","releaseId":1}}
May 12 23:55:50 c801556 resin-supervisor[1853]: [event]   Event: Service exit {"service":{"appId":1,"serviceId":1,"serviceName":"bleuberrypi","releaseId":1}}
May 12 23:55:51 c801556 resin-supervisor[1853]: [event]   Event: Service exit {"service":{"appId":1,"serviceId":4,"serviceName":"logspout","releaseId":1}}
May 12 23:55:51 c801556 resin-supervisor[1853]: [event]   Event: Service exit {"service":{"appId":1,"serviceId":2,"serviceName":"mongodb","releaseId":1}}
May 12 23:55:57 c801556 resin-supervisor[1853]: [debug]   Handling of local mode switch is completed
May 12 23:55:58 c801556 resin-supervisor[1853]: [event]   Event: Network creation {}
May 12 23:55:58 c801556 resin-supervisor[1853]: [event]   Event: Service install {"service":{"appId":1772693,"serviceId":894989,"serviceName":"wifi-connect","releaseId":1794299}}
May 12 23:56:00 c801556 resin-supervisor[1853]: [error]   Error from the API: 401
May 12 23:56:00 c801556 resin-supervisor[1853]: [error]   Non-200 response from the API! Status code: 401 - message: Error
May 12 23:56:00 c801556 resin-supervisor[1853]: [error]         at /usr/src/app/dist/app.js:22:554770
May 12 23:56:00 c801556 resin-supervisor[1853]: [error]       at runMicrotasks (<anonymous>)
May 12 23:56:00 c801556 resin-supervisor[1853]: [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
May 12 23:56:00 c801556 resin-supervisor[1853]: [error]       at async /usr/src/app/dist/app.js:22:554078
May 12 23:56:00 c801556 resin-supervisor[1853]: [error]       at async /usr/src/app/dist/app.js:22:555600
May 12 23:56:01 c801556 resin-supervisor[1853]: [event]   Event: Service installed {"service":{"appId":1772693,"serviceId":894989,"serviceName":"wifi-connect","releaseId":1794299}}
May 12 23:56:01 c801556 resin-supervisor[1853]: [event]   Event: Service start {"service":{"appId":1772693,"serviceId":894989,"serviceName":"wifi-connect","releaseId":1794299}}
May 12 23:56:03 c801556 resin-supervisor[1853]: [event]   Event: Service started {"service":{"appId":1772693,"serviceId":894989,"serviceName":"wifi-connect","releaseId":1794299}}
May 12 23:56:04 c801556 resin-supervisor[1853]: [debug]   Spawning journald with: chroot  /mnt/root journalctl -a -S 2021-05-12 23:56:04 -o json CONTAINER_ID_FULL=cea07aa950c185688e68a8eca67274ad9ff992475b46ebe124c1fc1dcfd8091b
May 12 23:56:05 c801556 resin-supervisor[1853]: [event]   Event: Service install {"service":{"appId":1772693,"serviceId":1005296,"serviceName":"logspout","releaseId":1794299}}
May 12 23:56:05 c801556 resin-supervisor[1853]: [event]   Event: Service install {"service":{"appId":1772693,"serviceId":834761,"serviceName":"mongodb","releaseId":1794299}}
May 12 23:56:06 c801556 resin-supervisor[1853]: [event]   Event: Service installed {"service":{"appId":1772693,"serviceId":1005296,"serviceName":"logspout","releaseId":1794299}}
May 12 23:56:06 c801556 resin-supervisor[1853]: [event]   Event: Service start {"service":{"appId":1772693,"serviceId":1005296,"serviceName":"logspout","releaseId":1794299}}
May 12 23:56:06 c801556 resin-supervisor[1853]: [event]   Event: Service installed {"service":{"appId":1772693,"serviceId":834761,"serviceName":"mongodb","releaseId":1794299}}
May 12 23:56:06 c801556 resin-supervisor[1853]: [event]   Event: Service start {"service":{"appId":1772693,"serviceId":834761,"serviceName":"mongodb","releaseId":1794299}}
May 12 23:56:10 c801556 resin-supervisor[1853]: [error]   Error from the API: 401
May 12 23:56:10 c801556 resin-supervisor[1853]: [error]   Non-200 response from the API! Status code: 401 - message: Error
May 12 23:56:10 c801556 resin-supervisor[1853]: [error]         at /usr/src/app/dist/app.js:22:554770
May 12 23:56:10 c801556 resin-supervisor[1853]: [error]       at runMicrotasks (<anonymous>)
May 12 23:56:10 c801556 resin-supervisor[1853]: [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
May 12 23:56:10 c801556 resin-supervisor[1853]: [error]       at async /usr/src/app/dist/app.js:22:554078
May 12 23:56:11 c801556 resin-supervisor[1853]: [event]   Event: Service started {"service":{"appId":1772693,"serviceId":1005296,"serviceName":"logspout","releaseId":1794299}}
May 12 23:56:11 c801556 resin-supervisor[1853]: [debug]   Spawning journald with: chroot  /mnt/root journalctl -a -S 2021-05-12 23:56:11 -o json CONTAINER_ID_FULL=65ada9faf873463bc9333192adcad82ce16f4dd530c6103bbd6ba0d2e5c25ddf
May 12 23:56:11 c801556 resin-supervisor[1853]: [event]   Event: Service started {"service":{"appId":1772693,"serviceId":834761,"serviceName":"mongodb","releaseId":1794299}}
May 12 23:56:12 c801556 resin-supervisor[1853]: [debug]   Cannot parse Volume: fa4ab2d6e975f12fcbc7afc76b9fc35875ead9de2b5a8542225dc21b58840015
May 12 23:56:12 c801556 resin-supervisor[1853]: [debug]   Cannot parse Volume: 272aeab0ad96f742ebcc87525b5081145b07b8aa0365ed9dc56f4c397fcdd5c1
May 12 23:56:12 c801556 resin-supervisor[1853]: [debug]   Spawning journald with: chroot  /mnt/root journalctl -a -S 2021-05-12 23:56:11 -o json CONTAINER_ID_FULL=d3b4f5beb2b04cb87df86ca39f61047f1515c7c9fdf6c60a90cb64650b4c2c1f
May 12 23:56:13 c801556 resin-supervisor[1853]: [debug]   Cannot parse Volume: fa4ab2d6e975f12fcbc7afc76b9fc35875ead9de2b5a8542225dc21b58840015
May 12 23:56:13 c801556 resin-supervisor[1853]: [debug]   Cannot parse Volume: 272aeab0ad96f742ebcc87525b5081145b07b8aa0365ed9dc56f4c397fcdd5c1
May 12 23:56:14 c801556 resin-supervisor[1853]: [event]   Event: Service install {"service":{"appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","releaseId":1794299}}
May 12 23:56:15 c801556 resin-supervisor[1853]: [event]   Event: Service installed {"service":{"appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","releaseId":1794299}}
May 12 23:56:15 c801556 resin-supervisor[1853]: [event]   Event: Service start {"service":{"appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","releaseId":1794299}}
May 12 23:56:18 c801556 resin-supervisor[1853]: [event]   Event: Service started {"service":{"appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","releaseId":1794299}}
May 12 23:56:19 c801556 resin-supervisor[1853]: [debug]   Spawning journald with: chroot  /mnt/root journalctl -a -S 2021-05-12 23:56:19 -o json CONTAINER_ID_FULL=6380e6fe26b7e69c50676d5b91ddd39433a3f873be1cfef00a351fc99629e55d
May 12 23:56:19 c801556 resin-supervisor[1853]: [debug]   Cannot parse Volume: fa4ab2d6e975f12fcbc7afc76b9fc35875ead9de2b5a8542225dc21b58840015
May 12 23:56:19 c801556 resin-supervisor[1853]: [debug]   Cannot parse Volume: 272aeab0ad96f742ebcc87525b5081145b07b8aa0365ed9dc56f4c397fcdd5c1
May 12 23:56:20 c801556 resin-supervisor[1853]: [debug]   Cannot parse Volume: fa4ab2d6e975f12fcbc7afc76b9fc35875ead9de2b5a8542225dc21b58840015
May 12 23:56:20 c801556 resin-supervisor[1853]: [debug]   Cannot parse Volume: 272aeab0ad96f742ebcc87525b5081145b07b8aa0365ed9dc56f4c397fcdd5c1
May 12 23:56:20 c801556 resin-supervisor[1853]: [error]   Error from the API: 401
May 12 23:56:20 c801556 resin-supervisor[1853]: [error]   Non-200 response from the API! Status code: 401 - message: Error
May 12 23:56:20 c801556 resin-supervisor[1853]: [error]         at /usr/src/app/dist/app.js:22:554770
May 12 23:56:20 c801556 resin-supervisor[1853]: [error]       at runMicrotasks (<anonymous>)
May 12 23:56:20 c801556 resin-supervisor[1853]: [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
May 12 23:56:20 c801556 resin-supervisor[1853]: [error]       at async /usr/src/app/dist/app.js:22:554078
May 12 23:56:20 c801556 resin-supervisor[1853]: [error]       at async /usr/src/app/dist/app.js:22:555600
May 12 23:56:20 c801556 resin-supervisor[1853]: [debug]   Finished applying target state
May 12 23:56:20 c801556 resin-supervisor[1853]: [success] Device state apply success
May 12 23:56:30 c801556 resin-supervisor[1853]: [error]   Error from the API: 401
May 12 23:56:30 c801556 resin-supervisor[1853]: [error]   Non-200 response from the API! Status code: 401 - message: Error
May 12 23:56:30 c801556 resin-supervisor[1853]: [error]         at /usr/src/app/dist/app.js:22:554770
May 12 23:56:30 c801556 resin-supervisor[1853]: [error]       at runMicrotasks (<anonymous>)
May 12 23:56:30 c801556 resin-supervisor[1853]: [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
May 12 23:56:30 c801556 resin-supervisor[1853]: [error]       at async /usr/src/app/dist/app.js:22:554078
May 12 23:56:30 c801556 resin-supervisor[1853]: [error]       at async /usr/src/app/dist/app.js:22:555600
May 12 23:56:40 c801556 resin-supervisor[1853]: [error]   Error from the API: 401
May 12 23:56:40 c801556 resin-supervisor[1853]: [error]   Non-200 response from the API! Status code: 401 - message: Error
May 12 23:56:40 c801556 resin-supervisor[1853]: [error]         at /usr/src/app/dist/app.js:22:554770
May 12 23:56:40 c801556 resin-supervisor[1853]: [error]       at runMicrotasks (<anonymous>)
May 12 23:56:40 c801556 resin-supervisor[1853]: [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
May 12 23:56:40 c801556 resin-supervisor[1853]: [error]       at async /usr/src/app/dist/app.js:22:554078
May 12 23:56:40 c801556 resin-supervisor[1853]: [error]       at async /usr/src/app/dist/app.js:22:555600
29

I am now experiencing this error on all 4 of my test devices, including one with a production-type build image.

The last 2 devices started the error at the exact same time - during the release of a new version, actually before the error I reported yesterday (I just hadn’t noticed it yet). Strangely enough, it seems to have happened before anything actually changed on-device: the error starts before the image download completes.

Here’s a resin_supervisor log from one of the two devices:

May 12 19:38:58 929f4c2  [api]     GET /v1/healthy 200 - 4.076 ms
May 12 19:43:58 929f4c2  [api]     GET /v1/healthy 200 - 5.091 ms
May 12 19:45:34 929f4c2  [debug]   Attempting container log timestamp flush...
May 12 19:45:34 929f4c2  [debug]   Container log timestamp flush complete
May 12 19:46:26 929f4c2   [event]   Event: Update notification {}
May 12 19:46:26 929f4c2  [api]     POST /v1/update 204 - 22.892 ms
May 12 19:46:27 929f4c2  [info]    Applying target state
May 12 19:46:27 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:46:27 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:46:27 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:46:27 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:46:27 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:46:27 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:46:27 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:46:27 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:46:27 929f4c2  [debug]   Replacing container for service bleuberrypi because of config changes:
May 12 19:46:27 929f4c2   [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41","workingDir":""}}
May 12 19:46:27 929f4c2  [debug]   Replacing container for service bleuberrypi because of config changes:
May 12 19:46:27 929f4c2   [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41","workingDir":""}}
May 12 19:46:27 929f4c2   [event]   Event: Delta image download {"image":{"name":"registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41","appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","imageId":3607352,"releaseId":1796667,"dependent":0,"dockerImageId":null}}
May 12 19:46:28 929f4c2  [debug]   delta([bleuberrypi] registry2.balena-cloud.com/v2/115619be84c203142e230edc6ad528d8@sha256:ef1f82d530c286f3f5d6d645820e6e579782424976e6623100960abe293d4335): Starting delta to registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41
May 12 19:46:28 929f4c2  [error]   Error from the API: 401
May 12 19:46:28 929f4c2  [error]   Non-200 response from the API! Status code: 401 - message: Error
May 12 19:46:28 929f4c2  [error]         at /usr/src/app/dist/app.js:22:554770
May 12 19:46:28 929f4c2  [error]       at runMicrotasks (<anonymous>)
May 12 19:46:28 929f4c2  [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
May 12 19:46:28 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:554078
May 12 19:46:28 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:555600
May 12 19:46:38 929f4c2  [error]   Error from the API: 401
May 12 19:46:38 929f4c2  [error]   Non-200 response from the API! Status code: 401 - message: Error
May 12 19:46:38 929f4c2  [error]         at /usr/src/app/dist/app.js:22:554770
May 12 19:46:38 929f4c2  [error]       at runMicrotasks (<anonymous>)
May 12 19:46:38 929f4c2  [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
May 12 19:46:38 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:554078
May 12 19:46:38 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:555600
May 12 19:46:48 929f4c2  [error]   Error from the API: 401
May 12 19:46:48 929f4c2  [error]   Non-200 response from the API! Status code: 401 - message: Error
May 12 19:46:48 929f4c2  [error]         at /usr/src/app/dist/app.js:22:554770
May 12 19:46:48 929f4c2  [error]       at runMicrotasks (<anonymous>)
May 12 19:46:48 929f4c2  [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
May 12 19:46:48 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:554078
May 12 19:46:48 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:555600
May 12 19:46:49 929f4c2   [event]   Event: Delta still processing remotely. {}
May 12 19:46:49 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:46:49 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:46:50 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:46:50 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:46:50 929f4c2  [debug]   Replacing container for service bleuberrypi because of config changes:
May 12 19:46:50 929f4c2   [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41","workingDir":""}}
May 12 19:46:50 929f4c2  [debug]   Replacing container for service bleuberrypi because of config changes:
May 12 19:46:50 929f4c2   [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41","workingDir":""}}
May 12 19:46:50 929f4c2   [event]   Event: Delta image download {"image":{"name":"registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41","appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","imageId":3607352,"releaseId":1796667,"dependent":0,"dockerImageId":null}}
May 12 19:46:50 929f4c2  [debug]   delta([bleuberrypi] registry2.balena-cloud.com/v2/115619be84c203142e230edc6ad528d8@sha256:ef1f82d530c286f3f5d6d645820e6e579782424976e6623100960abe293d4335): Starting delta to registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41
May 12 19:46:58 929f4c2  [error]   Error from the API: 401
May 12 19:46:58 929f4c2  [error]   Non-200 response from the API! Status code: 401 - message: Error
May 12 19:46:58 929f4c2  [error]         at /usr/src/app/dist/app.js:22:554770
May 12 19:46:58 929f4c2  [error]       at runMicrotasks (<anonymous>)
May 12 19:46:58 929f4c2  [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
May 12 19:46:58 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:554078
May 12 19:46:58 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:555600
May 12 19:47:01 929f4c2   [event]   Event: Delta still processing remotely. {}
May 12 19:47:01 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:47:01 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:47:02 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:47:02 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:47:03 929f4c2  [debug]   Replacing container for service bleuberrypi because of config changes:
May 12 19:47:03 929f4c2   [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41","workingDir":""}}
May 12 19:47:03 929f4c2  [debug]   Replacing container for service bleuberrypi because of config changes:
May 12 19:47:03 929f4c2   [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41","workingDir":""}}
May 12 19:47:03 929f4c2   [event]   Event: Delta image download {"image":{"name":"registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41","appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","imageId":3607352,"releaseId":1796667,"dependent":0,"dockerImageId":null}}
May 12 19:47:03 929f4c2  [debug]   delta([bleuberrypi] registry2.balena-cloud.com/v2/115619be84c203142e230edc6ad528d8@sha256:ef1f82d530c286f3f5d6d645820e6e579782424976e6623100960abe293d4335): Starting delta to registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41
May 12 19:47:04 929f4c2  [debug]   delta([bleuberrypi] registry2.balena-cloud.com/v2/115619be84c203142e230edc6ad528d8@sha256:ef1f82d530c286f3f5d6d645820e6e579782424976e6623100960abe293d4335): Applying balena delta...
May 12 19:47:04 929f4c2  [debug]   delta([bleuberrypi] registry2.balena-cloud.com/v2/115619be84c203142e230edc6ad528d8@sha256:ef1f82d530c286f3f5d6d645820e6e579782424976e6623100960abe293d4335): Using registry auth token
May 12 19:47:08 929f4c2  [error]   Error from the API: 401
May 12 19:47:08 929f4c2  [error]   Non-200 response from the API! Status code: 401 - message: Error
May 12 19:47:08 929f4c2  [error]         at /usr/src/app/dist/app.js:22:554770
May 12 19:47:08 929f4c2  [error]       at runMicrotasks (<anonymous>)
May 12 19:47:08 929f4c2  [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
May 12 19:47:08 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:554078
May 12 19:47:08 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:555600
May 12 19:47:11 929f4c2  [debug]   delta([bleuberrypi] registry2.balena-cloud.com/v2/115619be84c203142e230edc6ad528d8@sha256:ef1f82d530c286f3f5d6d645820e6e579782424976e6623100960abe293d4335): Delta applied successfully
May 12 19:47:11 929f4c2   [event]   Event: Image downloaded {"image":{"name":"registry2.balena-cloud.com/v2/735e2fd7a948d8a17d4c0c5cca0e883b@sha256:e9b5a8178c61e49a8ba92b7d7793d9154e3c8b8dd366bdd2bbd2687c73b9cc41","appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","imageId":3607352,"releaseId":1796667,"dependent":0,"dockerImageId":null}}
May 12 19:47:11 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:47:11 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:47:12 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:47:12 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:47:13 929f4c2  [debug]   Replacing container for service bleuberrypi because of config changes:
May 12 19:47:13 929f4c2   [debug]     Non-array fields:  {"added":{},"deleted":{},"updated":{"image":"sha256:334d6f7a9fac3443a0e693dfaec0487cdbeeab69fbca663bb0705b58293634e9","environment":{"BALENA_RELEASE_HASH":"f2affcf610eb8ddd7afeb2ccecf359b6"}}}
May 12 19:47:13 929f4c2  [debug]   Replacing container for service bleuberrypi because of config changes:
May 12 19:47:13 929f4c2   [debug]     Non-array fields:  {"added":{},"deleted":{},"updated":{"image":"sha256:334d6f7a9fac3443a0e693dfaec0487cdbeeab69fbca663bb0705b58293634e9","environment":{"BALENA_RELEASE_HASH":"f2affcf610eb8ddd7afeb2ccecf359b6"}}}
May 12 19:47:13 929f4c2   [event]   Event: Service kill {"service":{"appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","releaseId":1794299}}
May 12 19:47:13 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:47:13 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:47:14 929f4c2   [event]   Event: Service exit {"service":{"appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","releaseId":1794299}}
May 12 19:47:14 929f4c2   [event]   Event: Service stop {"service":{"appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","releaseId":1794299}}
May 12 19:47:14 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:47:14 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:47:14 929f4c2   [event]   Event: Image removal {"image":{"id":68,"name":"registry2.balena-cloud.com/v2/115619be84c203142e230edc6ad528d8@sha256:ef1f82d530c286f3f5d6d645820e6e579782424976e6623100960abe293d4335","appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","imageId":3600239,"releaseId":1794299,"dependent":0,"dockerImageId":"sha256:06aebe6106a6dec2a252ccf315c815058b1da4b21a367bca941f803c37c728b1"}}
May 12 19:47:14 929f4c2   [event]   Event: Image removed {"image":{"id":68,"name":"registry2.balena-cloud.com/v2/115619be84c203142e230edc6ad528d8@sha256:ef1f82d530c286f3f5d6d645820e6e579782424976e6623100960abe293d4335","appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","imageId":3600239,"releaseId":1794299,"dependent":0,"dockerImageId":"sha256:06aebe6106a6dec2a252ccf315c815058b1da4b21a367bca941f803c37c728b1"}}
May 12 19:47:14 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:47:14 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:47:16 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:47:16 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:47:16 929f4c2   [event]   Event: Service install {"service":{"appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","releaseId":1796667}}
May 12 19:47:16 929f4c2   [event]   Event: Service installed {"service":{"appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","releaseId":1796667}}
May 12 19:47:16 929f4c2   [event]   Event: Service start {"service":{"appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","releaseId":1796667}}
May 12 19:47:17 929f4c2   [event]   Event: Service started {"service":{"appId":1772693,"serviceId":833392,"serviceName":"bleuberrypi","releaseId":1796667}}
May 12 19:47:17 929f4c2  [debug]   Spawning journald with: chroot  /mnt/root journalctl -a -S 2021-05-12 19:47:17 -o json CONTAINER_ID_FULL=cc76aa6f935e22a4aeec261799e67bdad2af58a969788d336ee8ce14eb776d90
May 12 19:47:17 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:47:18 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:47:18 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:47:18 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:47:18 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:47:18 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:47:18 929f4c2  [debug]   Cannot parse Volume: f89f9b00d450a7f02ab5bee6f137d7d3b971bdd401f6fd27f7437a2b85e05492
May 12 19:47:18 929f4c2  [debug]   Cannot parse Volume: d1f8722e83dad4f9a09c458ec6ef5e3f55cc83d276cce9de8691b3cbfb104b9e
May 12 19:47:18 929f4c2  [error]   Error from the API: 401
May 12 19:47:18 929f4c2  [error]   Non-200 response from the API! Status code: 401 - message: Error
May 12 19:47:18 929f4c2  [error]         at /usr/src/app/dist/app.js:22:554770
May 12 19:47:18 929f4c2  [error]       at runMicrotasks (<anonymous>)
May 12 19:47:19 929f4c2  [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
May 12 19:47:19 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:554078
May 12 19:47:19 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:555600
May 12 19:47:19 929f4c2  [debug]   Finished applying target state
May 12 19:47:19 929f4c2  [success] Device state apply success
May 12 19:47:29 929f4c2  [error]   Error from the API: 401
May 12 19:47:29 929f4c2  [error]   Non-200 response from the API! Status code: 401 - message: Error
May 12 19:47:29 929f4c2  [error]         at /usr/src/app/dist/app.js:22:554770
May 12 19:47:29 929f4c2  [error]       at runMicrotasks (<anonymous>)
May 12 19:47:29 929f4c2  [error]       at processTicksAndRejections (internal/process/task_queues.js:97:5)
May 12 19:47:29 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:554078
May 12 19:47:29 929f4c2  [error]       at async /usr/src/app/dist/app.js:22:555600
30

Support access to the original device has been granted for another week. I can give access to one or two of the other devices also if that would be helpful.

32

Hi,

Thank you for re-enabling support access. I’ve been looking at the device this morning, and can see lots of odd symptoms, such as multiple versions of the supervisor image on the device. I’ve let the supervisor team know that the device is available again, and they will take a look when they can. If it’s OK with you, could you keep the device online and with support access, as this investigation may take a while, but it would be super valuable to know what’s happened/happening.

Thanks for this!
Phil

34

Hi @phil-d-wilson,
Yes of course, I will make sure the device remains online and support access remains enabled. Thank you for your time and investigation!

38

Hey, so I figured out what the issue was with device f7402c253897c86a51dbcae4e617d2ab and resolved it. The device firstly was affected by a known issue where the database file on device has incorrect state. This database file tracks various things and one of them being the images downloaded. We haven’t been able to identify the source for this issue but plan to resolve it by just removing that table as documented in Remove dependancy on 'image' table in the device database · Issue #1627 · balena-os/balena-supervisor · GitHub.

Because of this issue, the device was reporting a bunch of services being downloaded but they weren’t actually on the device anymore… Interestingly though, what I hadn’t seen before was that of those services “downloaded”, 2 of them mapped to release IDs which were not long available in the dashboard.I have asked some other folks to check into this some more.

Can you share your other device UUIDs and I’ll resolve them as well. It’s pretty rare to have seen this happen to so many of your devices tbh. I’ll be sure to try my best to prioritize a fix for this.

1 Like
40

I can confirm I do not see API errors anymore - it looks like you fixed it. Thank you! I was able to update the Supervisor and everything still seems to be working.

Interestingly though, what I hadn’t seen before was that of those services “downloaded”, 2 of them mapped to release IDs which were not long available in the dashboard.

I have definitely deleted some releases using the API, in case that helps explain this.

Here are the other device UUIDs (I’ve enabled support access on all of them):

  • 929f4c2b9796715af9fece92834b9e27
  • 0a1ff6b8d61985a7abec9e65e69faa1f
  • c801556fc2c8e380f425077adf10450b

Thanks again for looking into and fixing this! It’s great to know that - if necessary - it can be fixed in-place (i.e. doesn’t necessitate a re-flash of the device).