Hey @shaunmulligan, just jumping in on this bandwagon 
I think this is related to my earlier ticket (Container delta sizes very large despite all large layers using cache.). Not sure why I haven’t noticed until now, but as you describe, it does seem like the deltas that get generated and displayed are somehow based on a combination of popularity and what devices are currently online.
That being said, I’m not sure the previous release it uses is always the newest, in addition to being most popular with online devices. We currently have 2 devices online: the one I’m actively changing and one that’s running a release from 10 days ago. There are other devices running a newer (9 day old) release, but they’re not online. Every build I’ve made recently has generated deltas based on the 10 day old release.
Does that make sense for the 2 deltas you described? One from small one from the latest release before this new one - the one that will be pushed to my device under test - and one from the 10 day old release because that other device is the “second best choice among powered on devices”? Shouldn’t the printed deltas reflect the latest->new change, since that’s what I’m actively working on?
It would certainly be nice to see the actual delta size for the device I’m actually testing, but my bigger concern is that the quoted build times for each service are much smaller than the actual total build time. I think it might be related to uploading these large deltas that I know are never going to get used since I never intend to put my test build on more than just my one device. For example:
[Info] Release: 0348eb74dc5162f1d2fee387c65aafe9 (id: 1549883)
[Info] ┌─────────────┬────────────┬────────────┬──────────────────────┐
[Info] │ Service │ Image Size │ Delta Size │ Build Time │
[Info] ├─────────────┼────────────┼────────────┼──────────────────────┤
[Info] │ nautilus │ 310.96 MB │ 57.07 MB │ 32 seconds │
[Info] ├─────────────┼────────────┼────────────┼──────────────────────┤
[Info] │ nemo │ 264.84 MB │ 74.33 KB │ 1 minute, 42 seconds │
[Info] ├─────────────┼────────────┼────────────┼──────────────────────┤
[Info] │ ui-bridge │ 310.96 MB │ 57.07 MB │ 28 seconds │
[Info] ├─────────────┼────────────┼────────────┼──────────────────────┤
[Info] │ ntrip-proxy │ 310.96 MB │ 57.07 MB │ 34 seconds │
[Info] ├─────────────┼────────────┼────────────┼──────────────────────┤
[Info] │ eos │ 44.58 MB │ 4.18 MB │ 12 seconds │
[Info] └─────────────┴────────────┴────────────┴──────────────────────┘
[Info] Build finished in 7 minutes, 47 seconds
The actual change here was literally adding a comment to a Python file and changing the version.txt file that is copied into the container in the absolute last Dockerfile step. None of the services should have changed by more than a few KB, and eos shouldn’t have changed at all. Compared with that 10 day old release, however, they all changed dramatically.
Both generating image deltas and uploading images take a very long time. Of course, our images are rather large, but the actual Docker layers that were modified are quite small. All the big layers used cached values.
I’m not entirely sure I understand the benefit of automatically generating deltas for the “second best choice”. Why not just do it on demand when you actually update that device (or the fleet)?