Container delta sizes very large despite all large layers using cache.

For anyone who comes across this topic:

The issue with large delta sizes appears to be the result of the build system generating two sets of deltas - one to the most recently generated build, and another to the “most popular” build among currently powered-on devices. The delta sizes that get printed appear to be the latter set. This is being discussed further here: https://forums.balena.io/t/delta-size-not-shown-with-every-balena-push-to-the-cloud.

The slow builds issue seems to have been related to strain on the ARM build machines from other users’ builds. I still see it occasionally where a build will print Still Working... for anywhere from a few minutes to half an hour or more. Sometimes this happens before the build starts, and other times it happens suddenly in the middle of a build. Most of the time the build runs quickly though, at least until it starts generates deltas, at which point the “most popular” deltas can slow it down by quite a lot.

Cheers,

Adam