Failing to download image from openBalena

Product support balenaEngine
1

Quick one here: I’m using v10.11.0 of the supervisor currently so things are compatible with openBalena (self-hosted), and I’ve had issues with getting a device to pull a basic image and run it. There are a few odd errors, here’s the log from a failed build:

Sep 29 02:40:39 2f75674 resin-supervisor[3072]: [event]   Event: Update notification {}
Sep 29 02:40:39 2f75674 resin-supervisor[3072]: [api]     POST /v1/update 204 - 19.512 ms
Sep 29 02:40:47 2f75674 resin-supervisor[3072]: [error]   Failed to get target state for device: t: No delay specified in scheduledApply
Sep 29 02:40:47 2f75674 resin-supervisor[3072]: [event]   Event: Image download error {"error":{"message":"(HTTP code 404) no such image - no such image: registry.openbalena.local/v2/b552864b98f36dc0fc2b1fa870292b58@sha256:f33ec31c542b53edfe5485d010603fa88b5e3f36582e9d9a99f589d44cc658a5: No such image: registry.openbalena.local/v2/b552864b98f36dc0fc2b1fa870292b58@sha256:f33ec31c542b53edfe5485d010603fa88b5e3f36582e9d9a99f589d44cc658a5 ","stack":"Error: (HTTP code 404) no such image - no such image: registry.openbalena.local/v2/b552864b98f36dc0fc2b1fa870292b58@sha256:f33ec31c542b53edfe5485d010603fa88b5e3f36582e9d9a99f589d44cc658a5: No such image: registry.openbalena.local/v2/b552864b98f36dc0fc2b1fa870292b58@sha256:f33ec31c542b53edfe5485d010603fa88b5e3f36582e9d9a99f589d44cc658a5 \n    at /usr/src/app/dist/app.js:688:42813\n    at /usr/src/app/dist/app.js:688:42775\n    at m.buildPayload (/usr/src/app/dist/app.js:688:42785)\n    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:688:42277)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1125:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"image":{"name":"registry.openbalena.local/v2/b552864b98f36dc0fc2b1fa870292b58@sha256:f33ec31c542b53edfe5485d010603fa88b5e3f36582e9d9a99f589d44cc658a5","appId":5,"serviceId":1,"serviceName":"main","imageId":3,"releaseId":3,"dependent":0,"dockerImageId":null}}
Sep 29 02:40:49 2f75674 resin-supervisor[3072]: [event]   Event: Docker image download {"image":{"name":"registry.openbalena.local/v2/c4d2497c3acfa2f9fc4084f24486cb78@sha256:f33ec31c542b53edfe5485d010603fa88b5e3f36582e9d9a99f589d44cc658a5","appId":5,"serviceId":1,"serviceName":"main","imageId":4,"releaseId":4,"dependent":0,"dockerImageId":null}}

I’m using the latest openBalena, balenaOS v2.49.0, and the balena engine/supervisor is version 10.11.0. I’m not sure if any of those can change for compatibility purposes. If upgrading the supervisor will fix the issue I’ll give that a try, otherwise I am not sure what the issue is. Any ideas/help are much appreciated

4

Here is a sample of the output of balena logs as well, in case it is helpful:

[Logs]    [9/29/2020, 1:57:28 PM] Downloading image 'registry.openbalena.local/v2/a4082de564c94855b60ad98905656c64@sha256:38b453e08c08a022dc1113f1f6fae98c697aa42cd533ac8775d49dafd18d43a4'
[Logs]    [9/29/2020, 1:59:00 PM] Failed to download image 'registry.openbalena.local/v2/a4082de564c94855b60ad98905656c64@sha256:38b453e08c08a022dc1113f1f6fae98c697aa42cd533ac8775d49dafd18d43a4' due to '(HTTP code 404) no such image - no such image: registry.openbalena.local/v2/a4082de564c94855b60ad98905656c64@sha256:38b453e08c08a022dc1113f1f6fae98c697aa42cd533ac8775d49dafd18d43a4: No such image: registry.openbalena.local/v2/a4082de564c94855b60ad98905656c64@sha256:38b453e08c08a022dc1113f1f6fae98c697aa42cd533ac8775d49dafd18d43a4 '
[Logs]    [9/29/2020, 1:59:04 PM] Downloading image 'registry.openbalena.local/v2/a4082de564c94855b60ad98905656c64@sha256:38b453e08c08a022dc1113f1f6fae98c697aa42cd533ac8775d49dafd18d43a4'
[Logs]    [9/29/2020, 2:00:26 PM] Failed to download image 'registry.openbalena.local/v2/a4082de564c94855b60ad98905656c64@sha256:38b453e08c08a022dc1113f1f6fae98c697aa42cd533ac8775d49dafd18d43a4' due to '(HTTP code 404) no such image - no such image: registry.openbalena.local/v2/a4082de564c94855b60ad98905656c64@sha256:38b453e08c08a022dc1113f1f6fae98c697aa42cd533ac8775d49dafd18d43a4: No such image: registry.openbalena.local/v2/a4082de564c94855b60ad98905656c64@sha256:38b453e08c08a022dc1113f1f6fae98c697aa42cd533ac8775d49dafd18d43a4 '
[Logs]    [9/29/2020, 2:00:31 PM] Downloading image 'registry.openbalena.local/v2/a4082de564c94855b60ad98905656c64@sha256:38b453e08c08a022dc1113f1f6fae98c697aa42cd533ac8775d49dafd18d43a4'

The previous logs are directly from the device (resin-supervisor) itself, not sure which is more relevant for troubleshooting purposes.

edit: May be due to the version of openBalena being used. I tried to run curl -X "https://api.openbalena.local/v2/application(1)" and it gave a “cannot get” error, but v5 gives an unauthorized error. Is there documentation describing compatibility between openBalena/balenaEngine anywhere?

5

For some reason, manually restarting the supervisor (via systemctl stop resin-supervisor and then start) fixed this issue on the development device. Not sure why exactly this is caused by since manually restarting was all that was needed, though this isn’t really a great fix for production devices…

6

Hi @cball,
Happy to hear it has worked for you when restarting the supervisor.
We’ve encountered the Failed to download image error before but it can have serveral root causes. Do you think you can share the entire build log with us so that we can take a closer look and identify the issue?
Moreover, what kind of devices do you use?

Best,
Georgia

8

Currently I am using a custom device (not officially supported) which is similar to many other devices that are supported.

The resin-supervisor logs for the device look like the following:

Oct 05 23:34:14 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:34:14 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:34:14 2f75674 resin-supervisor[2042]: [event]   Event: Docker image download {"image":{"name":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3","appId":5,"serviceId":1,"serviceName":"main","imageId":8,"releaseId":8,"dependent":0,"dockerImageId":null}}
Oct 05 23:34:15 2f75674 resin-supervisor[2042]: [event]   Event: Service exit {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:34:21 2f75674 resin-supervisor[2042]: [info]    Internet Connectivity: OK
Oct 05 23:35:01 2f75674 resin-supervisor[2042]: [error]   LogBackend: server responded with status code: 504
Oct 05 23:35:26 2f75674 resin-supervisor[2042]: [event]   Event: Service restart {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:35:26 2f75674 resin-supervisor[2042]: [event]   Event: Service exit {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:35:42 2f75674 resin-supervisor[2042]: [event]   Event: Service restart {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:35:42 2f75674 resin-supervisor[2042]: [event]   Event: Service exit {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:36:15 2f75674 resin-supervisor[2042]: [event]   Event: Service restart {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:36:15 2f75674 resin-supervisor[2042]: [event]   Event: Service exit {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:36:16 2f75674 resin-supervisor[2042]: [error]   LogBackend: server responded with status code: 504
Oct 05 23:37:08 2f75674 resin-supervisor[2042]: [event]   Event: Service restart {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:37:46 2f75674 resin-supervisor[2042]: [event]   Event: Service exit {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:37:58 2f75674 resin-supervisor[2042]: [error]   LogBackend: server responded with status code: 504
Oct 05 23:38:00 2f75674 resin-supervisor[2042]: [event]   Event: Image download error {"error":{"message":"(HTTP code 404) no such image - no such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3: No such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3 ","stack":"Error: (HTTP code 404) no such image - no such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3: No such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3 \n    at /usr/src/app/dist/app.js:688:42813\n    at /usr/src/app/dist/app.js:688:42775\n    at m.buildPayload (/usr/src/app/dist/app.js:688:42785)\n    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:688:42277)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1125:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"image":{"name":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3","appId":5,"serviceId":1,"serviceName":"main","imageId":8,"releaseId":8,"dependent":0,"dockerImageId":null}}
Oct 05 23:38:02 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:38:02 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:38:02 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:38:02 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:38:02 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:38:02 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:38:02 2f75674 resin-supervisor[2042]: [event]   Event: Docker image download {"image":{"name":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3","appId":5,"serviceId":1,"serviceName":"main","imageId":8,"releaseId":8,"dependent":0,"dockerImageId":null}}
Oct 05 23:38:50 2f75674 resin-supervisor[2042]: [error]   LogBackend: server responded with status code: 504
Oct 05 23:39:04 2f75674 resin-supervisor[2042]: [api]     GET /v1/healthy 200 - 18.254 ms
Oct 05 23:39:12 2f75674 resin-supervisor[2042]: [event]   Event: Service restart {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:39:19 2f75674 resin-supervisor[2042]: [event]   Event: Service exit {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:40:02 2f75674 resin-supervisor[2042]: [error]   LogBackend: server responded with status code: 504
Oct 05 23:40:37 2f75674 resin-supervisor[2042]: [event]   Event: Service restart {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:40:39 2f75674 resin-supervisor[2042]: [event]   Event: Service exit {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:41:27 2f75674 resin-supervisor[2042]: [error]   LogBackend: server responded with status code: 504
Oct 05 23:41:29 2f75674 resin-supervisor[2042]: [event]   Event: Image download error {"error":{"message":"(HTTP code 404) no such image - no such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3: No such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3 ","stack":"Error: (HTTP code 404) no such image - no such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3: No such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3 \n    at /usr/src/app/dist/app.js:688:42813\n    at /usr/src/app/dist/app.js:688:42775\n    at m.buildPayload (/usr/src/app/dist/app.js:688:42785)\n    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:688:42277)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1125:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"image":{"name":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3","appId":5,"serviceId":1,"serviceName":"main","imageId":8,"releaseId":8,"dependent":0,"dockerImageId":null}}
Oct 05 23:41:32 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:41:32 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:41:32 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:41:32 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:41:32 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:41:32 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:41:32 2f75674 resin-supervisor[2042]: [event]   Event: Docker image download {"image":{"name":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3","appId":5,"serviceId":1,"serviceName":"main","imageId":8,"releaseId":8,"dependent":0,"dockerImageId":null}}
Oct 05 23:42:19 2f75674 resin-supervisor[2042]: [error]   LogBackend: server responded with status code: 504
Oct 05 23:43:10 2f75674 resin-supervisor[2042]: [event]   Event: Service restart {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:43:10 2f75674 resin-supervisor[2042]: [event]   Event: Service exit {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:44:00 2f75674 resin-supervisor[2042]: [error]   LogBackend: server responded with status code: 504
Oct 05 23:44:09 2f75674 resin-supervisor[2042]: [event]   Event: Service restart {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:44:09 2f75674 resin-supervisor[2042]: [event]   Event: Service exit {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:44:10 2f75674 resin-supervisor[2042]: [debug]   Attempting container log timestamp flush...
Oct 05 23:44:11 2f75674 resin-supervisor[2042]: [debug]   Container log timestamp flush complete
Oct 05 23:44:12 2f75674 resin-supervisor[2042]: [api]     GET /v1/healthy 200 - 13.323 ms
Oct 05 23:44:59 2f75674 resin-supervisor[2042]: [error]   LogBackend: server responded with status code: 504
Oct 05 23:45:04 2f75674 resin-supervisor[2042]: [event]   Event: Image download error {"error":{"message":"(HTTP code 404) no such image - no such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3: No such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3 ","stack":"Error: (HTTP code 404) no such image - no such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3: No such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3 \n    at /usr/src/app/dist/app.js:688:42813\n    at /usr/src/app/dist/app.js:688:42775\n    at m.buildPayload (/usr/src/app/dist/app.js:688:42785)\n    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:688:42277)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1125:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"image":{"name":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3","appId":5,"serviceId":1,"serviceName":"main","imageId":8,"releaseId":8,"dependent":0,"dockerImageId":null}}
Oct 05 23:45:09 2f75674 resin-supervisor[2042]: [event]   Event: Service restart {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:45:09 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:45:09 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:45:09 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:45:09 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:45:09 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:45:09 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:45:09 2f75674 resin-supervisor[2042]: [event]   Event: Docker image download {"image":{"name":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3","appId":5,"serviceId":1,"serviceName":"main","imageId":8,"releaseId":8,"dependent":0,"dockerImageId":null}}
Oct 05 23:45:55 2f75674 resin-supervisor[2042]: [error]   LogBackend: server responded with status code: 504
Oct 05 23:46:23 2f75674 resin-supervisor[2042]: [event]   Event: Service exit {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:47:08 2f75674 resin-supervisor[2042]: [event]   Event: Service restart {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:47:08 2f75674 resin-supervisor[2042]: [event]   Event: Service exit {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:47:13 2f75674 resin-supervisor[2042]: [error]   LogBackend: server responded with status code: 504
Oct 05 23:47:54 2f75674 resin-supervisor[2042]: [event]   Event: Service restart {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:47:54 2f75674 resin-supervisor[2042]: [event]   Event: Service exit {"service":{"appId":5,"serviceId":1,"serviceName":"main","releaseId":7}}
Oct 05 23:48:41 2f75674 resin-supervisor[2042]: [event]   Event: Image download error {"error":{"message":"(HTTP code 404) no such image - no such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3: No such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3 ","stack":"Error: (HTTP code 404) no such image - no such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3: No such image: registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3 \n    at /usr/src/app/dist/app.js:688:42813\n    at /usr/src/app/dist/app.js:688:42775\n    at m.buildPayload (/usr/src/app/dist/app.js:688:42785)\n    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:688:42277)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1125:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"image":{"name":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3","appId":5,"serviceId":1,"serviceName":"main","imageId":8,"releaseId":8,"dependent":0,"dockerImageId":null}}
Oct 05 23:48:42 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:48:42 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:48:42 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:48:42 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:48:42 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:48:42 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]   Updating failed, but there's another update scheduled immediately:  Error: Failed to apply state transition steps.  Steps:["fetch"]
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]         at t.<anonymous> (/usr/src/app/dist/app.js:575:146448)
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]       at /usr/src/app/dist/app.js:575:130484
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]       at Object.throw (/usr/src/app/dist/app.js:575:130589)
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]       at s (/usr/src/app/dist/app.js:575:129354)
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]   Device state apply error Error: Failed to apply state transition steps.  Steps:["fetch"]
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]         at t.<anonymous> (/usr/src/app/dist/app.js:575:146448)
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]       at /usr/src/app/dist/app.js:575:130484
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]       at Object.throw (/usr/src/app/dist/app.js:575:130589)
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]       at s (/usr/src/app/dist/app.js:575:129354)
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [info]    Applying target state
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]   Scheduling another update attempt in 2000ms due to failure:  Error: Failed to apply state transition steps.  Steps:["fetch"]
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]         at t.<anonymous> (/usr/src/app/dist/app.js:575:146448)
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]       at /usr/src/app/dist/app.js:575:130484
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]       at Object.throw (/usr/src/app/dist/app.js:575:130589)
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]       at s (/usr/src/app/dist/app.js:575:129354)
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]   Device state apply error Error: Failed to apply state transition steps.  Steps:["fetch"]
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]         at t.<anonymous> (/usr/src/app/dist/app.js:575:146448)
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]       at /usr/src/app/dist/app.js:575:130484
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]       at Object.throw (/usr/src/app/dist/app.js:575:130589)
Oct 05 23:48:43 2f75674 resin-supervisor[2042]: [error]       at s (/usr/src/app/dist/app.js:575:129354)
Oct 05 23:48:44 2f75674 resin-supervisor[2042]: [error]   LogBackend: server responded with status code: 504
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [info]    Applying target state
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [debug]   Replacing container for service main because of config changes:
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [debug]     Non-array fields:  {"added":{},"deleted":{"command":{},"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.openbalena.local/v2/20cddde4ea5b53e1c63a3162e1cf691d@sha256:6436e0383d4623d38fd102a194d459d9ec6a47a7fc90899aa1facb7c60bb00c3"}}
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [error]   Scheduling another update attempt in 4000ms due to failure:  Error: Failed to apply state transition steps.  Steps:["fetch"]
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [error]         at t.<anonymous> (/usr/src/app/dist/app.js:575:146448)
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [error]       at /usr/src/app/dist/app.js:575:130484
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [error]       at Object.throw (/usr/src/app/dist/app.js:575:130589)
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [error]       at s (/usr/src/app/dist/app.js:575:129354)
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [error]   Device state apply error Error: Failed to apply state transition steps.  Steps:["fetch"]
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [error]         at t.<anonymous> (/usr/src/app/dist/app.js:575:146448)
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [error]       at /usr/src/app/dist/app.js:575:130484
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [error]       at Object.throw (/usr/src/app/dist/app.js:575:130589)
Oct 05 23:48:45 2f75674 resin-supervisor[2042]: [error]       at s (/usr/src/app/dist/app.js:575:129354)
Oct 05 23:49:06 2f75674 resin-supervisor[4716]: resin_supervisor
Oct 05 23:49:10 2f75674 resin-supervisor[4886]: Error response from daemon: No such container: resin_supervisor
Oct 05 23:49:10 2f75674 resin-supervisor[4896]: active
Oct 05 23:49:11 2f75674 resin-supervisor[4897]: Error: No such object: resin_supervisor
Oct 05 23:49:12 2f75674 resin-supervisor[4897]: Error: No such container: resin_supervisor
Oct 05 23:49:14 2f75674 resin-supervisor[4897]: Starting system message bus: dbus.
Oct 05 23:49:14 2f75674 resin-supervisor[4897]:  * Starting Avahi mDNS/DNS-SD Daemon: avahi-daemon
Oct 05 23:49:14 2f75674 resin-supervisor[4897]:    ...done.
Oct 05 23:49:14 2f75674 resin-supervisor[4897]: Updating certificates in /etc/ssl/certs...
Oct 05 23:49:18 2f75674 resin-supervisor[4897]: 1 added, 0 removed; done.
Oct 05 23:49:18 2f75674 resin-supervisor[4897]: Running hooks in /etc/ca-certificates/update.d...
Oct 05 23:49:18 2f75674 resin-supervisor[4897]: done.
Oct 05 23:49:23 2f75674 resin-supervisor[4897]: [info]    Supervisor v10.11.0 starting up...
Oct 05 23:49:23 2f75674 resin-supervisor[4897]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Oct 05 23:49:23 2f75674 resin-supervisor[4897]: [debug]   Starting event tracker
Oct 05 23:49:23 2f75674 resin-supervisor[4897]: [debug]   Starting logging infrastructure
Oct 05 23:49:23 2f75674 resin-supervisor[4897]: [debug]   Starting api binder
Oct 05 23:49:23 2f75674 resin-supervisor[4897]: [event]   Event: Supervisor start {}
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [debug]   Performing database cleanup for container log timestamps
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [info]    Previous engine snapshot was not stored. Skipping cleanup.
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [debug]   Handling of local mode switch is completed
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [debug]   Connectivity check enabled: true
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [debug]   Starting periodic check for IP addresses
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [info]    Reporting initial state, supervisor version and API info
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [debug]   VPN status path exists.
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [info]    Waiting for connectivity...
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [debug]   Skipping preloading
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [info]    Starting API server
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [info]    Applying target state
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [debug]   Ensuring device is provisioned
Oct 05 23:49:24 2f75674 resin-supervisor[4897]: [debug]   Starting current state report
Oct 05 23:49:25 2f75674 resin-supervisor[4897]: [debug]   Supervisor API listening on allowed interfaces only

That is from running journalctl -u resin-supervisor --no-pager on a development image. Running balena logs (device) on the balena machine shows effectively the same as my earlier reply, consistently the device fails to download the image. I have found it at times is a 504 error, and other times I am getting the errors shown above (no such image)

11

Can you try manually pulling the image from inside the device? You’ll need to SSH in, do balena login registry.yourdomain.com and use d_<device-uuid> for username and the device API key for password (both taken from config.json in /mnt/boot/config.json). Then try balena pull <complete-image-reference-from-logs-above>.

13

I have tried to run balena login as you suggested but am getting an error:

Error response from daemon: Get https://registry.openbalena.local/v2/: error parsing HTTP 400 response body: invalid character ‘I’ looking for beginning of value: “Invalid scope”

The device UUID and API key were both simply as I had them in /mnt/boot/config.json. I also tried logging in as the main user of balena (which I normally balena login with when using balena-cli) but this failed too. Tried to pull directly without the login but of course got the error that it required a docker login, I assume the issue is there

14

Hi there,

Would you mind providing the logs from the registry service around the time when you tried that balena pull?

15

I double checked the services using systemctl list-unit-files and didn’t see any balena/resin ones with registry in the name running. Is it under a different name?

Currently running balenaOS v2.49.0 for reference

16

You’ll have to grab those logs from your openBalena server, not the device itself. The container should start with registry.

18

On the openBalena server the first message (after starting) when the device is turned on is:

error authorizing context: authorization token required (with some following text)

I get a similar issue when trying to manually balena pull the image via the url. It first has error authorizing context: authorization token required and follows this with another error (separate message just milliseconds after) with error authorizing context: insufficient scope

The GET requests are also a little strange, as they have the format "GET /v2/v2/(hash)/blobs/sha256:(hash)" - these follow the failed authorization.

Apologies for not directly sending the logs in full, I’ll see if I can get access to them in a nicer way if it would help

23

Hi @cball – thanks for the additional detail. We’re investigating this, and will get back to you with more information. A couple of questions in the meantime:

  • Do you have any non-custom devices that are registered with your openBalena server, and do they exhibit this problem? I’m trying to determine if this problem shows up only on the custom devices in particular.

  • For the log messages you post above, can you confirm that the messages are from the registry service?

  • Can you confirm that the registry_endpoint field in /mnt/boot/config.json looks correct? It should be just the hostname.

Thanks in advance,
Hugh

26

Can you please try running again ‘balena login registry.openbalena.local -u “d_${DEVICE_UUID}”’ from a shell into the device, pasting the output here as well as the respective logs from ‘./scripts/compose exec registry journalctl -fn100’ and ‘./scripts/compose exec api journalctl -fn100’

28

Hi Hugh,

Apologies for the later reply, currently working one day a week. I’ll try to get as much detail as possible today to debug

Do you have any non-custom devices that are registered with your openBalena server, and do they exhibit this problem? I’m trying to determine if this problem shows up only on the custom devices in particular.

Originally the same configuration I am using now was tested with Raspberry Pi 4s and this local configuration worked entirely as expected with openBalena - there have not been changes to the openBalena setup since those earlier tests. The openBalena version used currently is 2.0.3 for reference, and Raspberry Pi 4s were tested with it and worked w/this configuration locally

For the log messages you post above, can you confirm that the messages are from the registry service?

Yes, those messages were from the registry service on the server.

Can you confirm that the registry_endpoint field in /mnt/boot/config.json looks correct? It should be just the hostname.

Yes, the registryEndpoint value in /mnt/boot/config.json is the registry url - this is just a local IP on the device (eg 192.168.1.xxx) that is the same as vpnEngpoint and apiEndpoint values. One minor difference is that the apiEndpoint value includes the https:// in front.

Interestingly enough, after I did a full reboot of everything (server/device) and completely changed the image I was attempting to run to one of the examples (the python hello world one) everything has worked fine.

The image I was working with was using docker-compose and I am not 100% sure it was cross-compiling correctly for the architecture - that may have been part of the issue I was facing. I’ll be looking through today to be as sure as I can be that this current iteration I am working off of is functioning as expected. At this point pulling the container to the device and running correctly does appear to be working, though I will try to incrementally step up the complexity of the builds I am doing/images used to see if I encounter this issue again.

29

I still have found issues with this manual login - I am getting error responses from the server despite the device succeeding to pull & run the example as in my previous reply.

I ran balena login registry.openbalena.local -u "d_6599f480ad406d1a4c1a4ef64c5e0a43" and entered the password as the deviceApiKey from /mnt/boot/config.json and got this reponse:

Error response from daemon: Get https://registry.openbalena.local/v2/: error parsing HTTP 400 response body: invalid character 'I' looking for beginning of value: "Invalid scope".

It did not matter what I tried to use as a password, all had the same error.

I put the logs for the registry service /API on pastebin as they were otherwise far over the character limit:

registry pastebin: https://pastebin.com/LA5H6qdk
API logs pastebin: https://pastebin.com/sBGatjD6

31

Can you please run curl -i https://registry.openbalena.local/v2/ and paste the response headers here?

32

Yes, the registryEndpoint value in /mnt/boot/config.json is the registry url - this is just a local IP on the device (eg 192.168.1.xxx) that is the same as vpnEngpoint and apiEndpoint values

What do you mean by this? What is exactly the value of registryEndpoint?

34 
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

To answer the second question, the value of registryEndpoint is “registry.openbalena.local”. In /etc/hosts I have put in a line to map registry.openbalena.local to a local IP (192.168.1.100) which is the same IP as api.openbalena.local, s3.balena.local etc map to.

36

You can skip cert validation with the -k flag, so try curl -ik https://registry.openbalena.local/v2/

37

Hi,

Just following up from the earlier thread… Were you able to get a result by skipping the cert validation with curl -ik https://registry.openbalena.local/v2/ ?

John