Is there a setting or switch that needs to be enabled in order for the delta sizes to be shown at the end of a balena push to the cloud? I noticed that it sometimes shows delta sizes and sometimes it doesn’t. The application is for an ASUS TinkerBoard and I’m using the latest versions of the CLI and OS and Delta Updates are enabled in the Fleet Configuration. Any pointers would be greatly appreciated.
Hi. Does it show it generates deltas the times when it does not show delta sizes?
Hmm, I do not know. What does it say if it generates deltas (what message should I look for)? Also, if it doesn’t generate deltas, how can I change it to generate deltas?
Hi there,
Do you mean that only occasionally there is output concerning delta size, or that there used to be such output and now there is no such output?
Typically, the delta sizes are displayed as a separate column that is part of the service table.
There used to be a column that shows the delta size but recently this column is not shown anymore for subsequent pushes.
Great, thank you for that. I suspect this issue might be related to some recently deployed changes in our builder – as a sanity check, do you have any rough estimate when you first saw that column disappear?
Last time I saw the delta column was 2-3 days ago, and yesterday it disappeared (if my recollection is correct). But I was also pushing new builds to new apps, so that might confuse the issue a bit.
Awesome, thanks for that added context. We’re discussing this issue internally and will keep you updated!
Hello, what CLI version are you using?
It is 12.18.0.
Just tried with 12.21.0 and no delta size. Here is the output:
[Info] Building on arm03
[Info] Pulling previous images for caching purposes…
[Success] Successfully pulled cache images
[…]
[Info] Uploading images
[Success] Successfully uploaded images
[Info] Built on arm03
[Success] Release successfully created!
[Info] Release: 02c1ae5a46f927e796e1dcce282255c5 (id: 1536066)
[Info] ┌─────────────┬────────────┬────────────┐
[Info] │ Service │ Image Size │ Build Time │
[Info] ├─────────────┼────────────┼────────────┤
[Info] │ networking │ 483.68 MB │ 6 seconds │
[Info] ├─────────────┼────────────┼────────────┤
[Info] │ dashboard │ 805.02 MB │ 28 seconds │
[Info] ├─────────────┼────────────┼────────────┤
[Info] │ receiver │ 137.29 MB │ 8 seconds │
[Info] ├─────────────┼────────────┼────────────┤
[Info] │ watchdog │ 197.12 MB │ 6 seconds │
[Info] ├─────────────┼────────────┼────────────┤
[Info] │ folderwatch │ 137.30 MB │ 5 seconds │
[Info] ├─────────────┼────────────┼────────────┤
[Info] │ powermgmt │ 137.29 MB │ 5 seconds │
[Info] └─────────────┴────────────┴────────────┘
[Info] Build finished in 2 minutes, 37 seconds
In addition to my previous comment, I can now also confirm that the delta size is indeed sometimes shown. However, for most pushes I have done, it is not shown.
Hi, thanks for the additional feedback. we are looking into this
Do you have deltas enabled? Is any of your devices online? Logs above show that deltas were skipped – it’d say something along the lines of “Generating image deltas” otherwise.
Deltas are enabled but it is possible that none of the devices are online. Is it necessary for at least one device to be online?
Do you have deltas enabled on the application level, or just per device (you can see it in fleet configuration variable and device configuration variable respectively)?
Hi Maik, I just confirmed that indeed deltas will only be generated at build time if at least 1 device is online at the time of the build. I will raise this to the product team to see if there is a strong reason to not always generate the delta even if the devices are all offline, I can’t think of a very good reason not to do it always.
It’s enabled in the fleet configuration.
Yeah, from what I understand the build time deltas currently ignore the fleet wide config and evaluate the devices in the fleet that are online, this is done so that it because the builder will generate 2 deltas. #1 the delta between the previous version and the one that was just built and #2 the delta between the current build and the most “popular” release running on online devices. I’ll chat to the team to see if there is a way we can do something intelligent when all devices on the fleet are offline.
Hey @shaunmulligan, just jumping in on this bandwagon 
I think this is related to my earlier ticket (Container delta sizes very large despite all large layers using cache.). Not sure why I haven’t noticed until now, but as you describe, it does seem like the deltas that get generated and displayed are somehow based on a combination of popularity and what devices are currently online.
That being said, I’m not sure the previous release it uses is always the newest, in addition to being most popular with online devices. We currently have 2 devices online: the one I’m actively changing and one that’s running a release from 10 days ago. There are other devices running a newer (9 day old) release, but they’re not online. Every build I’ve made recently has generated deltas based on the 10 day old release.
Does that make sense for the 2 deltas you described? One from small one from the latest release before this new one - the one that will be pushed to my device under test - and one from the 10 day old release because that other device is the “second best choice among powered on devices”? Shouldn’t the printed deltas reflect the latest->new change, since that’s what I’m actively working on?
It would certainly be nice to see the actual delta size for the device I’m actually testing, but my bigger concern is that the quoted build times for each service are much smaller than the actual total build time. I think it might be related to uploading these large deltas that I know are never going to get used since I never intend to put my test build on more than just my one device. For example:
[Info] Release: 0348eb74dc5162f1d2fee387c65aafe9 (id: 1549883)
[Info] ┌─────────────┬────────────┬────────────┬──────────────────────┐
[Info] │ Service │ Image Size │ Delta Size │ Build Time │
[Info] ├─────────────┼────────────┼────────────┼──────────────────────┤
[Info] │ nautilus │ 310.96 MB │ 57.07 MB │ 32 seconds │
[Info] ├─────────────┼────────────┼────────────┼──────────────────────┤
[Info] │ nemo │ 264.84 MB │ 74.33 KB │ 1 minute, 42 seconds │
[Info] ├─────────────┼────────────┼────────────┼──────────────────────┤
[Info] │ ui-bridge │ 310.96 MB │ 57.07 MB │ 28 seconds │
[Info] ├─────────────┼────────────┼────────────┼──────────────────────┤
[Info] │ ntrip-proxy │ 310.96 MB │ 57.07 MB │ 34 seconds │
[Info] ├─────────────┼────────────┼────────────┼──────────────────────┤
[Info] │ eos │ 44.58 MB │ 4.18 MB │ 12 seconds │
[Info] └─────────────┴────────────┴────────────┴──────────────────────┘
[Info] Build finished in 7 minutes, 47 seconds
The actual change here was literally adding a comment to a Python file and changing the version.txt file that is copied into the container in the absolute last Dockerfile step. None of the services should have changed by more than a few KB, and eos shouldn’t have changed at all. Compared with that 10 day old release, however, they all changed dramatically.
Both generating image deltas and uploading images take a very long time. Of course, our images are rather large, but the actual Docker layers that were modified are quite small. All the big layers used cached values.
I’m not entirely sure I understand the benefit of automatically generating deltas for the “second best choice”. Why not just do it on demand when you actually update that device (or the fleet)?