Deploy Fails - Unable to extract image digest

Product support
1

I’m attempting to create a new release with a command that I’ve been using successfully for the last several years.

balena deploy edo/development --registry-secrets=../../credentials.yml --build --debug

Regardless of what I try, I’ve been getting Deploy failed with the following error…

Unable to extract image digest (content hash) from image upload progress stream for image:
registry2.balena-cloud.com/v2/d408ca4c4855811636fc5c3cefe699d9:latest

ExpectedError: Unable to extract image digest (content hash) from image upload progress stream for image:
registry2.balena-cloud.com/v2/d408ca4c4855811636fc5c3cefe699d9:latest
    at pushImage (C:\Program Files\balena-cli\client\build\utils\compose_ts.js:698:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Promise.all (index 1)
    at async inspectAndPushImage (C:\Program Files\balena-cli\client\build\utils\compose_ts.js:710:42)
    at async Promise.all (index 4)
    at async pushAndUpdateServiceImages (C:\Program Files\balena-cli\client\build\utils\compose_ts.js:739:9)
    at async pushServiceImages (C:\Program Files\balena-cli\client\build\utils\compose_ts.js:748:5)
    at async C:\Program Files\balena-cli\client\build\utils\compose_ts.js:789:17
    at async awaitInterruptibleTask (C:\Program Files\balena-cli\client\build\utils\helpers.js:283:16)
    at async deployProject (C:\Program Files\balena-cli\client\build\utils\compose_ts.js:787:13)
    at async DeployCmd.deployProject (C:\Program Files\balena-cli\client\build\commands\deploy\index.js:146:27)
    at async DeployCmd.run (C:\Program Files\balena-cli\client\build\commands\deploy\index.js:50:25)
    at async DeployCmd._run (C:\Program Files\balena-cli\client\node_modules\@oclif\core\lib\command.js:312:22)
    at async Config.runCommand (C:\Program Files\balena-cli\client\node_modules\@oclif\core\lib\config\config.js:435:25)
    at async run (C:\Program Files\balena-cli\client\node_modules\@oclif\core\lib\main.js:95:16)
    at async C:\Program Files\balena-cli\client\build\app.js:78:13

I’m running this on a Windows 11 laptop. I’ve completely uninstalled/reinstalled Docker, tried going back to HyperV from WSL 2, tried different networks, and all attempted have yielded the same error.

I see this similar post from a couple months ago, but the ‘solution’ was to use Linux, which to me isn’t a valid solution to the problem.

Any ideas?

Edit: If I use a Linux VM I’m able to deploy this release which to me rules out any configuration errors with the release itself.

2

Hello @adhilton first of all welome to the balena community!

Could you please confirm the version of the balena CLI that you are using in Windows?

Thanks!

3

Hi @mpous,

Thank you for the response. I’m currently using 20.2.3. I also tested with 20.1.2, which was previously working for me, but that also currently yields the same error.

Thanks!

4

hi @adhilton , what version of docker desktop are you running on windows and did you recently update it? It sounds like very recently docker has changed the default for docker to start using buildKit for their builds (BuildKit on Windows After Docker Update · Issue #4594 · moby/buildkit · GitHub) this I think breaks our CLI as the underlying library we use to interact with the docker engine doesn’t yet support buildkit. Could you check that buildkit is disabled in your docker engine settings and if it is not could you disable it. It should be possible to do this from the docker desktop engine settings page, see image below:

buildkit-disable
buildkit-disable897×291 18.4 KB

5

Hi @shaunmulligan1,

Thank you for looking into that. I’m currently using Docker Engine v27.5.1 with Docker Desktop v4.38.0.

I just tested with the following settings but that unfortunately yielded the same result.

{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": false,
  "features": {
    "buildKit": false
  }
}

Note that in your screenshot, you had buildKit: true. I assume that you meant false, but I went ahead and tested both ways (restarting everything after changes). Enabled and disabled yielded the same results.

6

I just got this as well, and I’m unable to deploy any builds at the moment. I tested against two different fleets, as well as testing clean rebuilds. The one thing that does occur to me is that I just updated my MacOS version, but I got a successful build out prior to this. I did also try with the buildKit change, but with no luck

2 Likes
7

Hi, I encounter the same issue while trying to deploy on macOS Sequoia 15.5 and Balena CLI 22.1.1

Were you able to solve this issue ?

Thanks for your help

8

I’m having the issue on macOS 15.5 and Balena CLI 22.4.1, using Docker Desktop 4.45.0

9

@shaunmulligan1 Any ideas here? It seems like other people are experiencing this as well. I still haven’t been able to solve this. My only solution has been to deploy through a Ubuntu VM. I switched to a new laptop, clean install of everything and still have the same problem.

10

We’ve been experiencing the same issue and done some investigations. Interestingly we have run the same Balena deploy command on two different CI runners with seemingly the same versions and yet one fails and the other doesn’t, we aren’t 100% sure what is differing.

However, what seems to be throwing the is the docker-progress which is used by the balena-cli. Specifically the tryExtractDisgestHash() method.

See: docker-progress/lib/index.ts at master · balena-io-modules/docker-progress · GitHub

We captured the push stream in the failing case, and we noticed that the format being outputted is:

{
  "progressDetail": {},
  "aux": {
    "manifestPushedInsteadOfIndex": true,
    "originalIndex": {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "digest": "sha256:3f63d4239c0261cd43305d8d1e8160e7731c6f012e8a375b66a976addb08e433",
      "size": 856
    },
    "selectedManifest": {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:81b69fbe21d65854333b11b111d1456666162321af566c244bb19c912fd8170c",
      "size": 8836,
      "platform": {"architecture": "arm64", "os": "linux"}
    }
  }
}

Note that evt.aux.Digest does not exist and we have now evt.aux.selectedManifest.digest

We don’t however actually know what causes Docker to use one format over another given we’ve tried disabling and enabling buildkit with no luck. We tested with seemingly identical versions on two different runners with differing outcomes:

- Docker Engine 29.0.2 (same API version)
- Buildx v0.30.0
- BuildKit v0.25.2
- balena-cli 22.5.5

We’ve not really had the time to look into this further, **we are also at this time (given the lack of further time we can dedicate to investigating) unable to justify whether a change to docker-process is the best course of action.
**
@shaunmulligan1 We have also tried disabling buildkit but also with no luck. Any hints here?