Error when deploying "unknown blob"

Product support
1

Hello! We’ve recently switched to pushing images to our private repository on dockerhub as opposed to using Balena to build the images from source.

Quite often I have encountered the following issue below when pushing the images to Balena using the newer approach above.

I’ve been some times successful of sorting this out by removing a handful of old releases, I can’t be certain this is directly correlated though.

Any idea of what is causing this issue?

[Info]             Starting build for <redacted>, user <redacted>
[Info]             Dashboard link: <redacted>
[Info]             Building on c617b98
[Info]             Pulling previous images for caching purposes...
[Success]          Successfully pulled cache images
[Info]             Uploading images
[Success]          Successfully uploaded images
[Error]            Some services failed to build:
[Error]              Service: vector
[Error]                Error: error pulling image configuration: download failed after attempts=1: unknown blob
[Info]             Built on c617b98
[Error]            Not deploying release.

Note: It almost appears as if it is a timing issue, trying the same deploy at a later point in time seems to work. Locally pulling the images never fail though,

2

Hello @Simontaga looks like the unknown blob means that one or more layers of the vector image is not found.

Could you please confirm that the image exist in the image repository?

Thanks!

3

@mpous I can confirm the image exists each time this has happened, what I notice is that if I wait a few minutes after the image is pushed to the docker registry/repo and only then deploying to Balena it works.

My gut feeling is that Balena doesn’t do a fresh pull from the registry, and I have to wait for some cache to expire. I’ve attempted to use balena push <fleet> --nocache but it appears to have no effect.

I can add that when we deploy to Balena we specify <our_image>:latest in the compose for each deploy right now. I’ve also encountered a case where deploying the same image with the same tag to Balena seems to use a cached version and doesn’t always pull the image again from our docker repo.

1 Like
4

Thanks for sharing this @Simontaga

The docker suggested approach is to not use :latest, but using exact version tags so that you end up with reproducible builds.

If you are using the exact version tag, rather than :latest, we would only expect you to face that issue in case you try to push to balena with a specific tag, before the actual image becomes available on dockerhub.

That would indeed add a cache entry that the image doesn’t exist. If you only push after checking w/ dockerhub’s api that the image is indeed there, then this shouldn’t be an issue.
If you still want to use :latest you will have to always use --nocache, since otherwise the builders might picked an older cached layer, which is also how local docker build works.

5

Thanks for the thorough response @mpous

I can share that --nocache doesn’t appear to have any effect in my case, meaning I still encounter the “unknown blob” issue if I push to Balena early on while still being able to pull the identical same image locally. (I have tried this on multiple occasions)

Anyhow, tagging the images properly is probably the best way to go.

Update: Switched to tagging the images uniquely per commit and referencing those in the push.

1 Like
6

I’ll have to re-open this as the issue persists.

The current procedure on our end is:

1. Each commit on our end will result in a uniquely tagged image being pushed to dockerhub
2. We use a script which checks and waits until the images are available
3. We push a release to Balena referencing the new images.

Although this does still quite frequently result in the unknown blob issue.

The --nocache flag has also been added to step #3 but doesn’t seem to yield any results.

I was able to reproduce the issue locally, and I do not think this is an issue on Balena’s side.

1 Like
7

Thanks for sharing this @Simontaga

Happy that you could find the root cause! Feel free to share your solution here for the community if you can :slight_smile:

8

What I ended up doing in our pipeline before pushing to Balena we attempt to pull the images from docker until successful. At first I had only checked if the manifest was available, but that was not enough.

This works okay. Although there appears to still sometimes be a gap where even if we are definitely able to pull the images ourselves successfully, Balena still gives us the unknown blob error. It’s good enough for now, but would be great if a better solution could be found .

1 Like