Build fails with error - no such image

hpgmiskin · June 12, 2020, 7:38am

We are continually facing issues with the Balena builder where the build reaches the end and then fails with an error saying there is no such image. The ellipses indicates a lot of missing lines in both the following examples.

[image-processor]   ---> Running in 894b68088ad4
[image-processor]  Removing intermediate container 894b68088ad4
[image-processor]   ---> f792c206a731
[image-processor]  Successfully built f792c206a731
...
[===========================================>       ] 87%
[Error]            An error occured: (HTTP code 404) no such image - no such image: f792c206a731: No such image: f792c206a731:latest
[Info]             Built on arm03
[Error]            Not deploying release.

Here is an example where the same commit is also failing to build due to the no such image error:

[redis]            Using cache
[redis]             ---> 925c21ebd27c
[redis]            Successfully built 925c21ebd27c
...
[Info]             Generating image deltas from release 6d89854fc96ef76070cc5afb630b5ac7e2cf65a8 (id: 1415647)
[Warning]          Failed to generate deltas due to an internal error; will be generated on-demand
[Info]             Uploading images

[>                                                  ] 1%
[=>                                                 ] 2%
[Error]            An error occured: (HTTP code 404) no such image - no such image: 925c21ebd27c: No such image: 925c21ebd27c:latest
[Info]             Built on arm03
[Error]            Not deploying release.

I have seen this issue occurring multiple times. Following the first occurrence all the services will build from the very beginning on the next push. This issue has then been seen to occur again and again. While writing this message I have just seen the third failed build for the same commit. The most recent two build have taken 20+ minutes each.

Interestingly the final failure had the no such image error for the hardware service but also an error that I have not seen before on the camera service failed to set parent. Perhaps there could be some relation as they both seem to be associated with how intermediate layers are stored and related.

[hardware]         Removing intermediate container 3e701d6ba0a8
[hardware]          ---> 8233d31a5081
[hardware]         Successfully built 8233d31a5081
...
[camera]            ---> Running in 938c48a974c7
[camera]           Removing intermediate container 938c48a974c7
[camera]            ---> 95b5de5bb532
[camera]           Step 32/40 : COPY start_cmd.sh start.sh ./
[camera]           failed to export image: failed to set parent sha256:95b5de5bb5320e4a50e832447cf84cdee2cbd92fb10a8d519a76a846c2e1eb29: unknown parent image ID sha256:95b5de5bb5320e4a50e832447cf84cdee2cbd92fb10a8d519a76a846c2e1eb29
[Info]             Uploading images
[Error]            An error occured: (HTTP code 404) no such image - no such image: 8233d31a5081: No such image: 8233d31a5081:latest
[Info]             Built on arm03
[Error]            Not deploying release.

Let me know what more information might be needed, I have full logs from every failed build which I can share. I will continue retrying this build in the meantime, however, in the past we have found that these issues are time dependant therefore the only resolve is to wait for a number of hours and try again.

sradevski · June 12, 2020, 8:10am

Hey, as I said in the other thread, we are currently investigating this, and we’ll get back to you as soon as it’s resolved.

hpgmiskin · June 12, 2020, 5:00pm

I thought I would provide an update on this, having spent some time experimenting. The issue remains regardless if I use balena push or git push and occurs both when pushing from my computer and when running in CircleCI.

I would like to share a previous post in which I identified the same issue. At the time I did not attempt to use balena push to resolve the problem. If it is helpful for debugging I can provide the output from DEBUG=1 balena push.

I think a contributing factor to experiencing the no such image error is the complexity of the Dockerfile. Although it can sometimes fail on even the most simple Dockerfile.

Please keep me updated on the status of investigation, I will be happy to provide more information where required.

saintaardvark · June 12, 2020, 7:58pm

Hi there – thanks for the additional information. We are definitely investigating this, and will update you when we have a resolution.

All the best,

Hugh

hpgmiskin · June 15, 2020, 9:54am

Minor update from over the weekend, no need to respond as this is purely observational.

On Friday my builds were reliably failing with “no such image” errors. I ran a number of builds on Saturday and Sunday with no issues. On Monday morning I ran a build early in the day and it completed successfully, however, my builds have now started failing again.

Perhaps the issue could be due to the load on the build servers or the number of concurrent builds?

hpgmiskin · June 16, 2020, 5:06pm

Hi @saintaardvark,

Has there been any update on this issue? It is still causing us lots of problems.

We have implemented an emulated deploy on our build servers in order to avoid using Balena build servers, however, this can take a very long time to build and has started costing us for the time taken on CircleCI.

Are there any alternatives? Could we host our own build server? Alternatively, do paid plans on Balena get priority treatment or isolated build servers? We would happily pay in order to get more reliable builds.

As you can likely tell this is very high priority for us. It was basically ground all our development to a halt since first facing this issue last week.

Thanks in advance

isabellel · June 16, 2020, 8:25pm

Hello! is there any update on this issue?
Looks like your system status is all good https://status.balena.io/ but I’m getting a lot of “fails with error - no such image” failures when building.

Thanks

samothx · June 16, 2020, 8:34pm

We are still waiting for a potential fix to be deployed. We will update you once that happens.
Regards Thomas

Sebi55 · June 17, 2020, 10:35pm

Got the same issue with an external image from Docker Hub. In this case postgres

20k-ultra · June 17, 2020, 10:37pm

Hey, this builder issue has been resolved and you should be able to give it another try. Let us know if it still persists.

Sebi55 · June 17, 2020, 10:50pm

Ok I received like some few minutes ago by just using “git push balena master” with this docker-compose file:
version: “2.1”

services:
  postgres:
    image: "postgres:12.3-alpine"
    network_mode: "host"
    restart: always
  pmsgate:
    build: ./
    network_mode: "host"
    ports:
      - "80:80"
    labels:
      io.balena.features.dbus: '1'
    environment:
      DBUS_SYSTEM_BUS_ADDRESS: "unix:path=/host/run/dbus/system_bus_socket"
      POSTGRES_PASSWORD: test
  node-red:
    image: "nodered/node-red:latest"
    network_mode: "host"
    environment:
      - TZ=Europe/Berlin
    ports:
      - "1880:1880"
    volumes:
      - nodereddata:/data
volumes:
  nodereddata:
    driver: local

I receive either no such image for node-red or postgres but inconsistent between these.
My device is the following although it wont help you much: 4aed22f5df0ee20bfbafc39aa2112d0e

Sebi55 · June 17, 2020, 10:55pm

Here an image with a test just now:

Sebi55 · June 18, 2020, 6:23am

Now it seems to work but yesterday evening I was unable to push a new image

hpgmiskin · June 18, 2020, 6:59am

Hi @samothx and @20k-ultra,

I have run a test build this morning and we are still experiencing this issue! Here are the relevant lines of output from out build:

[interface]        Removing intermediate container 5e20ecc5499e
[interface]         ---> 843b2ade0934
[interface]        Step 14/24 : FROM balenalib/raspberrypi3-node:12-stretch-run
[interface]         ---> 8d8ca644b0c8
...
[interface]        Step 19/24 : COPY --from=build /app/dist ./dist/
[interface]        invalid from flag value build: No such image: sha256:843b2ade093423d28d6b8da52269ef8612e91362f39cd1b3865779ce4ac24b4d
[Info]             Still Working...
[Info]             Still Working...
[Info]             Still Working...
[Info]             Still Working...
[Info]             Still Working...
[Info]             Still Working...
[Info]             Still Working...
...
[Info]             Uploading images
[Error]            An error occured: (HTTP code 404) no such image - no such image: ab59410f34e4: No such image: ab59410f34e4:latest
[Info]             Built on arm03
[Error]            Not deploying release.

As you can see in the above logs the base layer is created and stored, however, at a later point the image cannot be found by the builder.

Is there any more information I can provide to help get to the bottom of this?

Thanks

vbersier · June 18, 2020, 7:01am

Same here, tried over the last 2 hours (it’s a long build) and I got the same error at the very end of the otherwise successful build:

[Info]            Starting to build smsl-provision
[Info]            Starting to build smsl-inference
[Info]            Starting to build kiosk
[Info]            Starting to build smsl-daemon
[Info]            Starting to build smsl-screen
[Success]         Successfully built service kiosk
[Success]         Successfully built service smsl-provision
[Success]         Successfully built service smsl-daemon
[Success]         Successfully built service smsl-inference
[Info]            Uploading images
[Error]           An error occured: (HTTP code 404) no such image - no such image: dd34bf226122: No such image: dd34bf226122:latest 
[Info]            Built on arm01
[Error]           Not deploying release.

Interestingly, I have no log line that says “Successfully built service smsl-screen” (EDIT: nevermind this, I had a build error in this particular container)

georgiats · June 18, 2020, 7:56am

Hi there,
I see some of you managed to push and that’s great! But still, some of you struggle.
@hpgmiskin could you please run the following command and let me know if that works for you:
balena push myApp --nocache

The --nocache flag will initiate a fresh build and not use any cache from previous builds of this project.

hpgmiskin · June 18, 2020, 8:03am

Hi @georgiats,

Thanks for your quick response.

My most recent build which failed did not use any cached layer. In addition the image which produced the error was around half way through the build process. This mid layer image can be seen in the logs I have posted above.

Here is the start of the build output from the failed build, where no cached layers were used:

[camera]           Step 1/26 : FROM balenalib/raspberrypi3-python:3.7-stretch-build
[redis]            Step 1/6 : FROM arm32v7/redis:6-buster
[hardware]         Step 1/21 : FROM balenalib/raspberrypi3-python:3.7-stretch-build
[metrics]          Step 1/11 : FROM balenalib/raspberrypi3-debian-python:3.5.7-build
[proxy]            Step 1/10 : FROM balenalib/raspberrypi3-node:12-stretch
[controller]       Step 1/22 : FROM balenalib/raspberrypi3-node:12-stretch-build as build
[image-processor]  Step 1/24 : FROM balenalib/raspberrypi3-python:3.7-stretch
[interface]        Step 1/24 : FROM balenalib/raspberrypi3-node:12-stretch-build as build
[proxy]             ---> 8d8ca644b0c8
[proxy]            Step 2/10 : RUN npm install -g npm@latest
[proxy]             ---> Running in 0cd8f45cd40c
[camera]            ---> 362d4b3df6b8
[camera]           Step 2/26 : RUN install_packages bzip2
[camera]            ---> Running in 92f327efeff7
[hardware]          ---> 362d4b3df6b8
[hardware]         Step 2/21 : ENV UDEV 1
[image-processor]   ---> 9d934bfae18e
[image-processor]  Step 2/24 : RUN install_packages curl unzip openjdk-8-jre-headless
[hardware]          ---> Running in 93ed8c131220
[image-processor]   ---> Running in c1ec7c4803e7
[hardware]         Removing intermediate container 93ed8c131220
[hardware]          ---> 6b9a11efd237
[hardware]         Step 3/21 : RUN apt-get update     && apt-get install -y make libusb-dev     && apt-get clean     && rm -rf /var/lib/apt/lists/*
[camera]           Here are a few details about this Docker image (For more information please visit https://www.balena.io/docs/reference/base-images/base-images/):

If needed I can still run the command with no cache? Although I am fairly certain, given the above build, it will not make any difference.

Thanks

georgiats · June 18, 2020, 8:34am

Hey @hpgmiskin, I see. Could you please attach here the entire build log, so that we can investigate further the issue?

hpgmiskin · June 18, 2020, 8:39am

Hi @georgiats,

Would it be possible to share the full log through a private channel of communication?

If needed we would be happy to upgrade to a paid account in order to share privately. We have been in discussion with Alision Davis about this issue, and which payment plan would be suitable for us.

Thanks

georgiats · June 18, 2020, 8:42am

No worries @hpgmiskin, you can send me a PM here, and I’ll pass the logs to the team.