We are continually facing issues with the Balena builder where the build reaches the end and then fails with an error saying there is no such image. The ellipses indicates a lot of missing lines in both the following examples.
[image-processor] ---> Running in 894b68088ad4 [image-processor] Removing intermediate container 894b68088ad4 [image-processor] ---> f792c206a731 [image-processor] Successfully built f792c206a731 ... [===========================================> ] 87% [Error] An error occured: (HTTP code 404) no such image - no such image: f792c206a731: No such image: f792c206a731:latest [Info] Built on arm03 [Error] Not deploying release.
Here is an example where the same commit is also failing to build due to the no such image error:
[redis] Using cache [redis] ---> 925c21ebd27c [redis] Successfully built 925c21ebd27c ... [Info] Generating image deltas from release 6d89854fc96ef76070cc5afb630b5ac7e2cf65a8 (id: 1415647) [Warning] Failed to generate deltas due to an internal error; will be generated on-demand [Info] Uploading images [> ] 1% [=> ] 2% [Error] An error occured: (HTTP code 404) no such image - no such image: 925c21ebd27c: No such image: 925c21ebd27c:latest [Info] Built on arm03 [Error] Not deploying release.
I have seen this issue occurring multiple times. Following the first occurrence all the services will build from the very beginning on the next push. This issue has then been seen to occur again and again. While writing this message I have just seen the third failed build for the same commit. The most recent two build have taken 20+ minutes each.
Interestingly the final failure had the
no such image error for the hardware service but also an error that I have not seen before on the camera service
failed to set parent. Perhaps there could be some relation as they both seem to be associated with how intermediate layers are stored and related.
[hardware] Removing intermediate container 3e701d6ba0a8 [hardware] ---> 8233d31a5081 [hardware] Successfully built 8233d31a5081 ... [camera] ---> Running in 938c48a974c7 [camera] Removing intermediate container 938c48a974c7 [camera] ---> 95b5de5bb532 [camera] Step 32/40 : COPY start_cmd.sh start.sh ./ [camera] failed to export image: failed to set parent sha256:95b5de5bb5320e4a50e832447cf84cdee2cbd92fb10a8d519a76a846c2e1eb29: unknown parent image ID sha256:95b5de5bb5320e4a50e832447cf84cdee2cbd92fb10a8d519a76a846c2e1eb29 [Info] Uploading images [Error] An error occured: (HTTP code 404) no such image - no such image: 8233d31a5081: No such image: 8233d31a5081:latest [Info] Built on arm03 [Error] Not deploying release.
Let me know what more information might be needed, I have full logs from every failed build which I can share. I will continue retrying this build in the meantime, however, in the past we have found that these issues are time dependant therefore the only resolve is to wait for a number of hours and try again.