Hello Adam, thanks for your feedback. We have seen arm03 being overloaded these last couple of days and have been looking into it, so your findings make sense. I attached an issue to this support thread so that you will get notified once we get to the bottom of them.
Hey guys,
Just wanted to check in and see how things are going. This morning the arm03 builds are running at normal speed, and suddenly the delta sizes for my “only the version.txt file changed” containers are back to normal: 2 KB instead of 80 MB! Did anything change upstream that would have fixed this? The deltas were still huge 100% of the time as of last night.
Cheers,
Adam
Hello, the builder hosts several concurrent builds and at times, which can last up to a few hours, it can become less responsive or build times may increase significantly. We generally monitor its resources and adapt, but it is normal for build times to fluctuate.
For anyone who comes across this topic:
The issue with large delta sizes appears to be the result of the build system generating two sets of deltas - one to the most recently generated build, and another to the “most popular” build among currently powered-on devices. The delta sizes that get printed appear to be the latter set. This is being discussed further here: https://forums.balena.io/t/delta-size-not-shown-with-every-balena-push-to-the-cloud.
The slow builds issue seems to have been related to strain on the ARM build machines from other users’ builds. I still see it occasionally where a build will print Still Working... for anywhere from a few minutes to half an hour or more. Sometimes this happens before the build starts, and other times it happens suddenly in the middle of a build. Most of the time the build runs quickly though, at least until it starts generates deltas, at which point the “most popular” deltas can slow it down by quite a lot.
Cheers,
Adam