Hey Hugh,
I haven’t had time to pull together a test case yet. I’m hoping to do so soon, but in the meantime it seems like it’s mostly back to normal all of the sudden so I’m not sure if I’ll be able to replicate it very easily in the first place unfortunately.
Building for a different application and then going back to the normal application mostly resolved the super long build times. Most builds since then have been around 4-6 minutes, which has been somewhat typical historically, though some as long as maybe 10-12 minutes, which is a bit longer than expected and than we’d like ideally. Still nothing near the 40+ minutes I was seeing earlier.
It seems much of the time is spent in the “generating image deltas” step after the containers have been built. For example:
[Info] Generating image deltas from release 8e75b83f1ac8300222555928e66a3528 (id: 1519144)
[Info] Still Working...
[Info] Still Working...
[Success] Successfully generated image deltas
[Info] Uploading images
[Success] Successfully uploaded images
[Info] Built on arm03
[Success] Release successfully created!
[Info] Release: 807513d16d11c6d529c22ec15f6fb822 (id: 1520140)
[Info] ┌─────────────┬────────────┬────────────┬───────────────────────┐
[Info] │ Service │ Image Size │ Delta Size │ Build Time │
[Info] ├─────────────┼────────────┼────────────┼───────────────────────┤
[Info] │ nautilus │ 309.82 MB │ 49.46 MB │ 2 minutes, 47 seconds │
[Info] ├─────────────┼────────────┼────────────┼───────────────────────┤
[Info] │ nemo │ 261.13 MB │ 1.53 KB │ 43 seconds │
[Info] ├─────────────┼────────────┼────────────┼───────────────────────┤
[Info] │ ui-bridge │ 309.82 MB │ 49.46 MB │ 2 minutes, 54 seconds │
[Info] ├─────────────┼────────────┼────────────┼───────────────────────┤
[Info] │ ntrip-proxy │ 309.82 MB │ 49.46 MB │ 2 minutes, 53 seconds │
[Info] ├─────────────┼────────────┼────────────┼───────────────────────┤
[Info] │ eos │ 44.76 MB │ 5.58 MB │ 2 seconds │
[Info] └─────────────┴────────────┴────────────┴───────────────────────┘
[Info] Build finished in 9 minutes, 11 seconds
This isn’t entirely crazy since the deltas are somewhat large (the tar file in the Nautilus container is pretty big), but still a bit surprising to me that it would take 5 or 6 minutes to generate even a 50 MB delta.
It also seems like the small delta sizes are working again today. You can see above that Nemo was only 1 KB instead of 35 MB, and Nautilus was 50 MB instead of the 77 MB it was consistently before. I’m not sure what changed there. I was still seeing large deltas after changing applications away and back, but suddenly it’s back to mostly normal.
When I get a chance I’ll try to put together a test repo if I can replicate the issue at all. Meanwhile, I hope there’s something in some of the server logs somewhere that might be helpful in some way.
Cheers,
Adam