Incorrect Delta Updates

Product support
1

I’m building releases for balena devices through balena cloud. The release consists of a docker-compose file that references an image in a private registry. Example:

version: '2.1'

services:
    someservice:
        image: registry.gitlab.com/my-image:5.1.3

Within this image in the private registry, we create a version file to identify the image. If I pull registry.gitlab.com/my-image:5.1.3 locally, run the image with a shell, and print the version file, I get “5.1.3” as expected.

However, when I build this image into a release using balena cloud, update a device to the release, open a shell into the service, then print the version file, I get “5.1.0”, which is the oldest released version of this image and is not the image I requested.

I’ve tried referencing the image in my docker-compose file using it’s sha identifier, which should be completely unique and immutable unlike a tag. That did not work either and exhibited the same failing behavior.

I also tried pushing the release with the --nocache option appended to the balena push command. That did not work either and exhibited the same failing behavior.

What could be going on here? I’m completely lost as to how this is possible.

  • Are there any known bad interactions with private registries hosted on Gitlab?
  • Is there any other caching behavior that could be interfering with this?
2

More context:

We also include version information in the image metadata (labels)

In the balena root OS, I inspected the image related to this service and saw that it does indeed have the correct labels. However, upon inspecting the version file inside, it’s still incorrect.

How can I pull this image locally and receive the correct contents but I can’t do the same on balena? Possibly a conflict with the delta updates?

3

I think I’ve narrowed this down to the delta update process

After purging a device and installing a new release from scratch, everything works as expected.

Is there a way to turn off delta updates? They don’t seem to work as expected, and while I appreciate the space/bandwidth saving characteristics of the delta updates, I care more that my software is correct :slight_smile:

4

Or is there a workaround here?

I can’t just keep purging my devices, or customers lose their data. Is there a way that I can at least clear the currently stored images to force a re-download? Running balena kill $(balena ps -aq) && balena image rm -f $(balena image ls -aq) (stop all running containers, remove all images) does not work.

5

Hello, what mechanism are you using to build/deploy a release to balenaCloud (e.g. git, balena-cli) and what are the exact commands you are running? Also, can you please share the build log output.