Failing to download image delta: no such file or directory. Broken image delta.

Product support
1

Prior to pushing a new release I deleted some of the test releases that were no longer used. I found a thread on this forum describing how to do this with a curl to https://api.balena-cloud.com/v6/release API. I am mentioning this because it feels related to the problem I am having.

After pushing the new release, I am failing to download an image delta:

Downloading delta for image 'registry2.balena-cloud.com/v2/44b757933e5cda7f07d430227148a4c6@sha256:2d9bb32a42dd40913ee4d5d370d79a1dab12711ac3ab43f091c9d1d1b792f838'
Failed to download image 'registry2.balena-cloud.com/v2/44b757933e5cda7f07d430227148a4c6@sha256:2d9bb32a42dd40913ee4d5d370d79a1dab12711ac3ab43f091c9d1d1b792f838' due to 'failed to get digest sha256:d0b01ad7854dad329240089c2e8894e8813610ebdaf87274ee44af3bbf3fc999: open /var/lib/docker/image/overlay2/imagedb/content/sha256/d0b01ad7854dad329240089c2e8894e8813610ebdaf87274ee44af3bbf3fc999: no such file or directory'

A more verbose log from balena-supervisor:

Dec 17 12:46:13 e7474a7 balena-supervisor[6753]: [debug]   delta([cashier] registry2.balena-cloud.com/v2/aff5b12bcde25bb3c9bd9f7c94972fbd@sha256:93d396d2b153ebf3865736245baa6801624b6be299ae08a1f4e559fab8c42b53): Delta failed with Error: failed to get digest sha256:d0b01ad7854dad329240089c2e8894e8813610ebdaf87274ee44af3bbf3fc999: open /var/lib/docker/image/overlay2/imagedb/content/sha256/d0b01ad7854dad329240089c2e8894e8813610ebdaf87274ee44af3bbf3fc999: no such file or directory
Dec 17 12:46:13 e7474a7 balena-supervisor[6753]: [event]   Event: Image download error {"error":{"message":"failed to get digest sha256:d0b01ad7854dad329240089c2e8894e8813610ebdaf87274ee44af3bbf3fc999: open /var/lib/docker/image/overlay2/imagedb/content/sha256/d0b01ad7854dad329240089c2e8894e8813610ebdaf87274ee44af3bbf3fc999: no such file or directory","stack":"Error: failed to get digest sha256:d0b01ad7854dad329240089c2e8894e8813610ebdaf87274ee44af3bbf3fc999: open /var/lib/docker/image/overlay2/imagedb/content/sha256/d0b01ad7854dad329240089c2e8894e8813610ebdaf87274ee44af3bbf3fc999: no such file or directory\n    at Stream.<anonymous> (/usr/src/app/dist/app.js:10:2307907)\n    at Stream.emit (events.js:310:20)\n    at drain (/usr/src/app/dist/app.js:2:304229)\n    at Stream.stream.queue.stream.push (/usr/src/app/dist/app.js:2:304636)\n    at Parser.parser.onToken (/usr/src/app/dist/app.js:10:255724)\n    at Parser.proto.write (/usr/src/app/dist/app.js:10:703427)\n    at Stream.<anonymous> (/usr/src/app/dist/app.js:10:254188)\n    at Stream.stream.write (/usr/src/app/dist/app.js:2:304505)\n    at IncomingMessage.ondata (_stream_readable.js:695:22)\n    at IncomingMessage.emit (events.js:310:20)\n    at addChunk (_stream_readable.js:286:12)\n    at readableAddChunk (_stream_readable.js:268:9)\n    at IncomingMessage.Readable.push (_stream_readable.js:209:10)\n    at HTTPParser.parserOnBody (_http_common.js:132:24)\n    at Socket.socketOnData (_http_client.js:476:22)\n    at Socket.emit (events.js:310:20)"},"image":{"name":"registry2.balena-cloud.com/v2/198bb4c45c329e388d5689fbb811f0ad@sha256:6f8fb5a534819274781bfa1b397152c5136fb1b71ddaa7fe0147bf00f24726e5","appId":1875486,"serviceId":1347202,"serviceName":"cashier","imageId":4355837,"releaseId":2014621,"dependent":0,"dockerImageId":null}}

I can see that indeed the digest is missing at /var/lib/docker/image/overlay2/imagedb/content/sha256/d0b01ad7854dad329240089c2e8894e8813610ebdaf87274ee44af3bbf3fc999. How can I force a re-download of this or what other steps one should take to fix this broken state?

2

Hi @alexm,

Can you try running with --nocache option? More details: balena CLI Documentation - Balena Documentation

Also, share the command you ran. And let us know if you into any more errors.

-N

4

I did try the nocache flag in the hopes that it would cause the images to be pulled again. But that didn’t change the errors about the missing digest.

The command I ran to deploy the image was the usual

balena deploy my-fleet --registry-secrets my-secrets.yml --multi-dockerignore

After that I tried the same but with --nocache

balena deploy my-fleet --registry-secrets my-secrets.yml --multi-dockerignore --nocache

Which created another release and queued it as a target release for my device. Any advice on resolving the errors would be much appreciated!

5

I am currently trying to manually pull the missing image, but not sure what docker auth token I need to use for registry2.balena-cloud.com.

balena pull registry2.balena-cloud.com/v2/44b757933e5cda7f07d430227148a4c6@sha256:2d9bb32a42dd40913ee4d5d370d79a1dab12711ac3ab43f091c9d1d1b792f838
Error response from daemon: pull access denied for registry2.balena-cloud.com/v2/44b757933e5cda7f07d430227148a4c6, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Edit: Used my balena username with an API key.

6

You can try some something like this to troubleshoot further:

deviceApiKey=$(cat /mnt/boot/config.json | jq -r .deviceApiKey)
uuid=$(cat /mnt/boot/config.json | jq -r .uuid)
endpoint=$(cat /mnt/boot/config.json | jq -r .registryEndpoint)

balena login ${endpoint} -u d_${uuid} -p ${deviceApiKey}

balena pull registry2.balena-cloud.com/v2/...

Alternatively, you can tryrm -rf /mnt/data/resin-data/balena-supervisor/database.sqlite and restart the supervisor. The nuclear option is to stop balena service and remove /var/lib/docker/*, but that may cause additional complications.

1 Like
9

For me the rm approach worked out:

rm -rf /mnt/data/resin-data/balena-supervisor/database.sqlite && systemctl restart balena-supervisor
10

My issue is also solved using this

rm -rf /mnt/data/resin-data/balena-supervisor/database.sqlite && systemctl restart balena-supervisor

My question is why i need to do this? why not balena superviser do itself to fix and download release version ?

11

For some older balenaOS (version 2.58.3+rev1, e.g Beaglebone Green), I had to run:

rm -rf /mnt/data/resin-data/resin-supervisor/database.sqlite && systemctl restart resin-supervisor

This “failed to download image” happened because I probably (but not sure) deleted a running release while downloading a new delta was in progress.