Experiencing Slow Build

‘[electron] Step 4/7 : COPY ./app ./
[electron] —> ff173bf084a3
[electron] Removing intermediate container a9bbecabd9b3
[electron] Step 5/7 : RUN npm install
[electron] —> Running in 7be492a7a144
[Info] Still working…
[Info] Still working…
[Info] Still working…
[Info] Still working…
[Info] Still working…’

Experiencing slow builds for arm devices

HI, can you please provide the full build log please? Also is this still an issue on your side?

Just to add:

  • Occasionally, if a native ARM server is unavailable due to load or maintenance, an x86 builder is used with QEMU emulation and this can result in a slower build. (We have a pool of native ARM servers that normally meet the demand for them.)
  • If you suspect that the native ARM servers are getting in the way rather than speeding up your builds, you can force a QEMU emulated build on x86 servers with the commands:
    • git push balena master:balena-emulated
    • balena push ... --emulated
    • balena build ... --emulated
    • balena deploy ... --emulated
  • Prefixing balena commands with DEBUG=1 adds debugging messages to the output logs:
    DEBUG=1 balena push ...

Me too, but only on arm01. That is, builds are running fine on arm03.

This is what I’m getting right now:

[Info]     Starting build for tlprototyping, user danilo_aghemo
[Info]     Dashboard link: https://dashboard.balena-cloud.com/apps/1109673/devices
[Info]     Building on arm01
[Info]     Pulling previous images for caching purposes...
[Success]  Successfully pulled cache images
[Info]     Still working..

Is it safe (for you) to just CTRL-C, so that maybe next time I will end up on a different system?

Yes it will interrupt the build and it is safe, but we currently have a known issue that we’re working on (and it’s not trivial unfortunately) where the first commit after an interrupted build could fail. If that happens, don’t worry and just push the commit again because it shouldn’t be stored on our side

Thanks for reporting this @daghemo. We had observed intermittent native ARM builder issues that eventually escalated to an incident, now resolved: https://status.balena.io/incidents/yfyfmc0dxw52

@wwalker, @daghemo, I wanted to also share an alternative to the ARM cloud builders or to emulators, which is to build on a local ARM device like a Raspberry Pi 3B (running a recent version of balenaOS). Enable local mode on the device, then run the commands below on a terminal on your laptop/desktop, replacing ‘192.168.2.3’ with your local device’s IP address:

# build the docker image on the device at the given IP address
balena build -a my_app -h 192.168.2.3 -p 2375 .
# copy the docker image from the device to the laptop/desktop
ssh root@192.168.2.3 -p 22222 balena-engine save my_image | docker load
# upload the image to the balena cloud for deployment to devices
balena deploy my_app my_image

The documentation for the build and deploy commands can be read with balena help build and balena help deploy, or on this page: https://www.balena.io/docs/reference/cli/#build-source-

The ssh command line above copies the image to the laptop’s / deskotp’s local docker daemon, which therefore must be installed and running. The images can be listed with the docker ps command. The counterpart command to copy an image in the opposite direction (from the laptop/desktop to a balena device) is:

docker save my_image | ssh root@192.168.2.3 -p 22222 balena-engine load
1 Like

@pdcastro, I’m going to try the suggested setup in the next few days.
Maybe arm01 is still having some problems:

[main]     E: Failed to fetch http://deb.debian.org/debian/dists/stretch/main/binary-armhf/Packages  Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device)
[main]     E: Failed to fetch http://deb.debian.org/debian/dists/stretch-updates/main/binary-armhf/Packages  Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device)
[main]     E: Failed to fetch http://security.debian.org/debian-security/dists/stretch/updates/main/binary-armhf/Packages  Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device)
[main]     E: Failed to fetch http://archive.raspberrypi.org/debian/dists/stretch/main/binary-armhf/Packages  Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device) [IP: 93.93.128.230 80]
[main]     E: Failed to fetch http://archive.raspbian.org/raspbian/dists/stretch/contrib/binary-armhf/Packages  Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device)
[main]     E: Some index files failed to download. They have been ignored, or old ones used instead.

@daghemo, we haven’t been able to reproduce the No space left on device errors, so I think it was a temporary issue. I have updated one of my posts above to list git and balena commands to force an emulated cloud build on x86 servers instead of using the native ARM servers, if you find that they are getting in the way rather than speeding up the build:

@pdcastro, our applications have also been getting the No space left on device errors on arm01 today, 4 hours or so after your reply

Thanks those commands though, I’m running a balena-emulated branch build now. We were previously using the balena-nocache remote branch but things seem to be building OK so far.

Thanks @pdcastro!
Now I’m having a new error from arm01:

[main]     Step 25/71 : ENV UDEV on
[main]     Using cache
[main]      ---> 9cf325d34ba8
[main]     Step 26/71 : ENV INITSYSTEM on
[main]     Using cache
[main]      ---> d434e5ff1fbd
[main]     Step 27/71 : ENV container docker
[main]     unable to find image "sha256:1937a5dd9d24b82834ac1a3143369a0247bcd4f6b3c82159ee3783e77e7f4468"
[Info]     Uploading images
[Success]  Successfully uploaded images
[

For the no space left error - there was an issue with one of our ARM builders - see https://status.balena.io/incidents/hjhvb0cf8pm7 and https://status.balena.io/incidents/yfyfmc0dxw52 - this should be resolved now.

@daghemo, if you’re still seeing the ‘unable to find image’ error, try bypassing the builder cache:

  • git push balena master:balena-nocache
  • balena push ... --nocache
  • balena build ... --nocache
  • balena deploy ... --nocache

No longer experiencing issues.

1 Like

Today i am also facing similar issue

Device - jetson nano ,

Tried 3-4 times building but same issue for around 30-40 minutes.

OS is production so can not use local mode.

Hi
I can confirm that at the moment we aren’t having any issues on our builders (https://status.balena.io/)
Are you still facing issues? Do you have complete logs of when you were trying to deploy?