image fails to download 404 no such image

jose_barrero · February 5, 2020, 11:15pm

I have an OpenBalena server running on ec2 and configured to store the registry files on S3
i am receiving this error almost every time a new device joins an application and tries to download the current version of the app.

Failed to download image ‘registry.xxxxx.com/v2/xxxxxxxxxx@sha256:a6xxxxxxxxxxxxxx’ due to '(HTTP code 404) no such image - no such image

But the image exists on the registry

The problem solves once I re-deploy the application, is this a normal behavior?
im using BalenaOS 2.46.1+rev1_prod for raspberrypi 3

dt-rush · February 5, 2020, 11:30pm

How are you verifying that the image exists in the registry?

jose_barrero · February 5, 2020, 11:58pm

on the S3 bucket im going through the files
@dt-rush do you know how can i do it using the api?

sradevski · February 6, 2020, 12:39pm

You might also get 404 if the device doesn’t have access to the image. I would assume the bucket is private? How are you setting the credentials/access token for the device to pull the image from the registry?

jose_barrero · February 6, 2020, 1:30pm

hi @sradevski. The weird thing is that sometimes it works just fine… so theres no permissions issues.

sradevski · February 6, 2020, 1:40pm

Is the bucket private? If so, how do you set the credentials? It’s just an assumption, but there might be a race condition between setting the environment variables and pulling the image, so any additional information would help.

jose_barrero · February 6, 2020, 1:48pm

yes the bucket is private. I just created a secret/access key that im passing to this env variables on OpenBalena

OPENBALENA_S3_ACCESS_KEY
OPENBALENA_S3_SECRET_KEY

sradevski · February 6, 2020, 3:00pm

If you could enable support access and share the uuid of a device that failed to fetch the image I will have a look for the possible cause for it.

sradevski · February 6, 2020, 3:05pm

My bad, I forgot this is about openBalena, so you cannot give me support access :). Can you look at the device logs and see what you can find out around the time when the device failed to fetch the image?

jose_barrero · February 6, 2020, 3:25pm

It seems like i deployed a new release faster than the devices downloading the last one, after waiting some time the updates are working as expected.

Ereski · February 6, 2020, 5:13pm

Glad it was resolved

Langhalsdino · March 3, 2020, 11:34am

I have a similar issue that i can not resolve

The download seems to break every 2 hours. Its a large update on a very slow connection.

[logs] [3/3/2020, 6:31:06 AM] Failed to download image ‘registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5’ due to '(HTTP code 404) no such image - no such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5: No such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5 ’
[Logs] [3/3/2020, 6:31:09 AM] Downloading image ‘registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5’
[Logs] [3/3/2020, 8:27:16 AM] Failed to download image ‘registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5’ due to '(HTTP code 404) no such image - no such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5: No such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5 ’
[Logs] [3/3/2020, 8:27:21 AM] Downloading image ‘registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5’
[Logs] [3/3/2020, 10:26:07 AM] Failed to download image ‘registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5’ due to '(HTTP code 404) no such image - no such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5: No such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5 ’
[Logs] [3/3/2020, 10:26:16 AM] Downloading image ‘registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5’
[Logs] [3/3/2020, 12:18:31 PM] Failed to download image ‘registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5’ due to '(HTTP code 404) no such image - no such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5: No such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5 ’
[Logs] [3/3/2020, 12:18:47 PM] Downloading image ‘registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5’

karaxuna · March 3, 2020, 11:45am

Can you ssh into the device and see supervisor logs (and send it to us)?

journalctl -f -a -u resin-supervisor --no-pager

Also can you check that device is not run out of disk space?

df -h

Langhalsdino · March 3, 2020, 11:49am

– Logs begin at Sat 2000-01-01 01:00:31 UTC. –
Mar 03 11:24:42 4296c1e resin-supervisor[2057]: [debug] Container log timestamp flush complete
Mar 03 11:25:28 4296c1e resin-supervisor[2057]: [api] GET /v1/healthy 200 - 3.110 ms
Mar 03 11:30:29 4296c1e resin-supervisor[2057]: [api] GET /v1/healthy 200 - 4.631 ms
Mar 03 11:34:42 4296c1e resin-supervisor[2057]: [debug] Attempting container log timestamp flush…
Mar 03 11:34:42 4296c1e resin-supervisor[2057]: [debug] Container log timestamp flush complete
Mar 03 11:35:29 4296c1e resin-supervisor[2057]: [api] GET /v1/healthy 200 - 9.577 ms
Mar 03 11:40:30 4296c1e resin-supervisor[2057]: [api] GET /v1/healthy 200 - 3.247 ms
Mar 03 11:44:42 4296c1e resin-supervisor[2057]: [debug] Attempting container log timestamp flush…
Mar 03 11:44:42 4296c1e resin-supervisor[2057]: [debug] Container log timestamp flush complete
Mar 03 11:45:30 4296c1e resin-supervisor[2057]: [api] GET /v1/healthy 200 - 3.535 ms

Filesystem                      Size  Used Avail Use% Mounted on
devtmpfs                        1.8G     0  1.8G   0% /dev
/dev/mmcblk0p13                 461M  314M  119M  73% /mnt/sysroot/active
/dev/disk/by-label/resin-state   19M  227K   17M   2% /mnt/state
overlay                         461M  314M  119M  73% /
tmpfs                           2.0G     0  2.0G   0% /dev/shm
tmpfs                           2.0G   14M  2.0G   1% /run
tmpfs                           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs                           2.0G     0  2.0G   0% /tmp
/dev/mmcblk0p12                  80M  1.8M   79M   3% /mnt/boot
tmpfs                           2.0G   32K  2.0G   1% /var/volatile
/dev/mmcblk0p16                  58G  9.6G   45G  18% /mnt/data

The raw docker image should be around 8 GB
Furthermore i use a fresh build of balenaOS 2.47.0+rev7 on the Nvidia Jetson Nano

tmigone · March 3, 2020, 12:32pm

Hi, thanks for providing more details. By the looks of it, the supervisor logs you shared are from a later period of time and not from the moment the issue was present. We need to check the supervisor logs for anything suspicious at around the time when the 404s start happening. If the device was restarted you’ll probably need to trigger the issue again to capture it.

Langhalsdino · March 3, 2020, 2:55pm

i will monitor them around 404 times and get back to you in 1 h

Langhalsdino · March 3, 2020, 4:04pm

@tmigone

Mar 03 13:00:38 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 3.883 ms
Mar 03 13:04:42 4296c1e resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 13:04:42 4296c1e resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 13:05:39 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 3.561 ms
Mar 03 13:10:39 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 8.310 ms
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [event]   Event: Image download error {"error":{"message":"(HTTP code 404) no such image - no such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5: No such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5 ","stack":"Error: (HTTP code 404) no such image - no such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5: No such image: registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5 \n    at /usr/src/app/dist/app.js:727:42130\n    at /usr/src/app/dist/app.js:727:42093\n    at m.buildPayload (/usr/src/app/dist/app.js:727:42103)\n    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:727:41603)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1125:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"image":{"name":"registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","appId":3,"serviceId":68,"serviceName":"myapp","imageId":135,"releaseId":135,"dependent":0,"dockerImageId":null}}
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]   Scheduling another update attempt in 32000ms due to failure:  Error: Failed to apply state transition steps.  Steps:["fetch"]
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]         at /usr/src/app/dist/app.js:614:16375
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at c (/usr/src/app/dist/app.js:9:77523)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at d (/usr/src/app/dist/app.js:312:329886)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at p (/usr/src/app/dist/app.js:312:329825)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at runCallback (timers.js:705:18)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at tryOnImmediate (timers.js:676:5)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at processImmediate (timers.js:658:5)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]   Device state apply error Error: Failed to apply state transition steps.  Steps:["fetch"]
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]         at /usr/src/app/dist/app.js:614:16375
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at c (/usr/src/app/dist/app.js:9:77523)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at d (/usr/src/app/dist/app.js:312:329886)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at p (/usr/src/app/dist/app.js:312:329825)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at runCallback (timers.js:705:18)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at tryOnImmediate (timers.js:676:5)
Mar 03 13:12:18 4296c1e resin-supervisor[2057]: [error]       at processImmediate (timers.js:658:5)
Mar 03 13:12:50 4296c1e resin-supervisor[2057]: [info]    Applying target state
Mar 03 13:12:50 4296c1e resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 13:12:50 4296c1e resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 13:12:50 4296c1e resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 13:12:50 4296c1e resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 13:12:50 4296c1e resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 13:12:50 4296c1e resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 13:12:50 4296c1e resin-supervisor[2057]: [event]   Event: Docker image download {"image":{"name":"registry.balena.me.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","appId":3,"serviceId":68,"serviceName":"myapp","imageId":135,"releaseId":135,"dependent":0,"dockerImageId":null}}
Mar 03 13:13:08 4296c1e resin-supervisor[2057]: [error]   LogBackend: server responded with status code: 504
Mar 03 13:14:42 4296c1e resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 13:14:42 4296c1e resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 13:15:40 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 4.571 ms
Mar 03 13:20:40 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 10.270 ms
Mar 03 13:24:42 4296c1e resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 13:24:42 4296c1e resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 13:25:41 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 3.564 ms
Mar 03 13:30:41 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 3.481 ms
Mar 03 13:34:42 4296c1e resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 13:34:42 4296c1e resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 13:35:42 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 4.566 ms
Mar 03 13:40:42 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 4.047 ms
Mar 03 13:44:42 4296c1e resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 13:44:42 4296c1e resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 13:45:43 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 5.430 ms
Mar 03 13:50:43 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 4.407 ms
Mar 03 13:54:42 4296c1e resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 13:54:42 4296c1e resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 13:55:44 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 4.238 ms
Mar 03 14:00:44 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 5.864 ms
Mar 03 14:04:42 4296c1e resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 14:04:42 4296c1e resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 14:05:45 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 3.387 ms
Mar 03 14:10:46 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 3.313 ms
Mar 03 14:14:42 4296c1e resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 14:14:42 4296c1e resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 14:15:46 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 5.311 ms
Mar 03 14:20:47 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 3.187 ms
Mar 03 14:24:42 4296c1e resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 14:24:42 4296c1e resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 14:25:47 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 3.345 ms
Mar 03 14:30:48 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 4.598 ms
Mar 03 14:34:42 4296c1e resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 14:34:42 4296c1e resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 14:35:48 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 4.268 ms
Mar 03 14:40:49 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 4.460 ms
Mar 03 14:44:42 4296c1e resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 14:44:42 4296c1e resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 14:45:49 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 4.314 ms
Mar 03 14:50:50 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 7.920 ms
Mar 03 14:54:42 4296c1e resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 14:54:42 4296c1e resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 14:55:50 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 6.126 ms
Mar 03 15:00:51 4296c1e resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 5.321 ms

Langhalsdino · March 3, 2020, 4:25pm

Further logs:

Mar 03 05:24:42  resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 05:24:42  resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 05:24:50  resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 5.768 ms
Mar 03 05:29:50  resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 8.583 ms
Mar 03 05:31:06  resin-supervisor[2057]: [event]   Event: Image download error {"error":{"message":"(HTTP code 404) no such image - no such image: registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5: No such image: registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5 ","stack":"Error: (HTTP code 404) no such image - no such image: registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5: No such image: registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5 \n    at /usr/src/app/dist/app.js:727:42130\n    at /usr/src/app/dist/app.js:727:42093\n    at m.buildPayload (/usr/src/app/dist/app.js:727:42103)\n    at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:727:41603)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1125:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"image":{"name":"registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","appId":3,"serviceId":68,"serviceName":"myapp","imageId":135,"releaseId":135,"dependent":0,"dockerImageId":null}}
Mar 03 05:31:07  resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 05:31:07  resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 05:31:07  resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 05:31:07  resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 05:31:07  resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 05:31:07  resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 05:31:07  resin-supervisor[2057]: [error]   Updating failed, but there's another update scheduled immediately:  Error: Failed to apply state transition steps.  Steps:["fetch"]
Mar 03 05:31:07  resin-supervisor[2057]: [error]         at /usr/src/app/dist/app.js:614:16375
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at c (/usr/src/app/dist/app.js:9:77523)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at d (/usr/src/app/dist/app.js:312:329886)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at p (/usr/src/app/dist/app.js:312:329825)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at runCallback (timers.js:705:18)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at tryOnImmediate (timers.js:676:5)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at processImmediate (timers.js:658:5)
Mar 03 05:31:07  resin-supervisor[2057]: [error]   Device state apply error Error: Failed to apply state transition steps.  Steps:["fetch"]
Mar 03 05:31:07  resin-supervisor[2057]: [error]         at /usr/src/app/dist/app.js:614:16375
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at c (/usr/src/app/dist/app.js:9:77523)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at d (/usr/src/app/dist/app.js:312:329886)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at p (/usr/src/app/dist/app.js:312:329825)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at runCallback (timers.js:705:18)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at tryOnImmediate (timers.js:676:5)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at processImmediate (timers.js:658:5)
Mar 03 05:31:07  resin-supervisor[2057]: [info]    Applying target state
Mar 03 05:31:07  resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 05:31:07  resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 05:31:07  resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 05:31:07  resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 05:31:07  resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 05:31:07  resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 05:31:07  resin-supervisor[2057]: [error]   Scheduling another update attempt in 2000ms due to failure:  Error: Failed to apply state transition steps.  Steps:["fetch"]
Mar 03 05:31:07  resin-supervisor[2057]: [error]         at /usr/src/app/dist/app.js:614:16375
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at c (/usr/src/app/dist/app.js:9:77523)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at d (/usr/src/app/dist/app.js:312:329886)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at p (/usr/src/app/dist/app.js:312:329825)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at runCallback (timers.js:705:18)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at tryOnImmediate (timers.js:676:5)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at processImmediate (timers.js:658:5)
Mar 03 05:31:07  resin-supervisor[2057]: [error]   Device state apply error Error: Failed to apply state transition steps.  Steps:["fetch"]
Mar 03 05:31:07  resin-supervisor[2057]: [error]         at /usr/src/app/dist/app.js:614:16375
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at c (/usr/src/app/dist/app.js:9:77523)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at d (/usr/src/app/dist/app.js:312:329886)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at p (/usr/src/app/dist/app.js:312:329825)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at runCallback (timers.js:705:18)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at tryOnImmediate (timers.js:676:5)
Mar 03 05:31:07  resin-supervisor[2057]: [error]       at processImmediate (timers.js:658:5)
Mar 03 05:31:09  resin-supervisor[2057]: [info]    Applying target state
Mar 03 05:31:09  resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 05:31:09  resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 05:31:09  resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 05:31:09  resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 05:31:09  resin-supervisor[2057]: [debug]   Replacing container for service myapp because of config changes:
Mar 03 05:31:09  resin-supervisor[2057]: [debug]     Non-array fields:  {"added":{},"deleted":{"entrypoint":{},"environment":{},"labels":{}},"updated":{"image":"registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","workingDir":""}}
Mar 03 05:31:09  resin-supervisor[2057]: [event]   Event: Docker image download {"image":{"name":"registry.balena.myapp.de/v2/1f5036873524ef1ae144d41f5f1697f4@sha256:7a3f64266578d6b938c396e92f31c6e04f409c5551f8a222438831aea46ff3a5","appId":3,"serviceId":68,"serviceName":"myapp","imageId":135,"releaseId":135,"dependent":0,"dockerImageId":null}}
Mar 03 05:31:56  resin-supervisor[2057]: [error]   LogBackend: server responded with status code: 504
Mar 03 05:34:42  resin-supervisor[2057]: [debug]   Attempting container log timestamp flush...
Mar 03 05:34:42  resin-supervisor[2057]: [debug]   Container log timestamp flush complete
Mar 03 05:34:51  resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 4.465 ms
Mar 03 05:39:51  resin-supervisor[2057]: [api]     GET /v1/healthy 200 - 4.229 ms

dtischler · March 3, 2020, 8:30pm

Hi Frederic, hope you are well! Can you do me a favor, and let us know which version of the Supervisor you are running?

You can determine it from those same ‘resin-supervisor’ logs you have been looking at…For example on a device of mine, scanning those logs, I see:

Mar 03 20:24:41 1ebb690 resin-supervisor[1921]: [info] Supervisor v10.6.27 starting up…

Thanks!
David

Langhalsdino · March 3, 2020, 11:16pm

I am running v10.6.27, too.

Topic		Replies	Views
Image randomly fails to download openBalena	6	545	August 26, 2019
OpenBalena API unavailable after pointing to external S3 bucket for registry storage openBalena support , docker , network	3	263	September 29, 2023
Failed to download image ... on 127.0.0.2:53: no such host openBalena	6	854	July 11, 2019
Getting started - Cannot Push Images to registry openBalena support	7	552	July 28, 2022
Device failing to download images - Error processing tar file(exit status 1): unexpected EOF' openBalena	2	4434	January 8, 2019

image fails to download 404 no such image

Related topics