Experiencing Slow Build

wwalker · February 11, 2019, 7:54pm

‘[electron] Step 4/7 : COPY ./app ./
[electron] —> ff173bf084a3
[electron] Removing intermediate container a9bbecabd9b3
[electron] Step 5/7 : RUN npm install
[electron] —> Running in 7be492a7a144
[Info] Still working…
[Info] Still working…
[Info] Still working…
[Info] Still working…
[Info] Still working…’

Experiencing slow builds for arm devices

majorz · February 12, 2019, 9:02am

HI, can you please provide the full build log please? Also is this still an issue on your side?

pdcastro · February 12, 2019, 5:39pm

Just to add:

Occasionally, if a native ARM server is unavailable due to load or maintenance, an x86 builder is used with QEMU emulation and this can result in a slower build. (We have a pool of native ARM servers that normally meet the demand for them.)
If you suspect that the native ARM servers are getting in the way rather than speeding up your builds, you can force a QEMU emulated build on x86 servers with the commands:
- git push balena master:balena-emulated
- balena push ... --emulated
- balena build ... --emulated
- balena deploy ... --emulated
Prefixing balena commands with DEBUG=1 adds debugging messages to the output logs:
DEBUG=1 balena push ...

daghemo · February 12, 2019, 6:15pm

Me too, but only on arm01. That is, builds are running fine on arm03.

daghemo · February 12, 2019, 7:34pm

This is what I’m getting right now:

[Info]     Starting build for tlprototyping, user danilo_aghemo
[Info]     Dashboard link: https://dashboard.balena-cloud.com/apps/1109673/devices
[Info]     Building on arm01
[Info]     Pulling previous images for caching purposes...
[Success]  Successfully pulled cache images
[Info]     Still working..

Is it safe (for you) to just CTRL-C, so that maybe next time I will end up on a different system?

thundron · February 12, 2019, 8:35pm

Yes it will interrupt the build and it is safe, but we currently have a known issue that we’re working on (and it’s not trivial unfortunately) where the first commit after an interrupted build could fail. If that happens, don’t worry and just push the commit again because it shouldn’t be stored on our side

pdcastro · February 12, 2019, 9:10pm

Thanks for reporting this @daghemo. We had observed intermittent native ARM builder issues that eventually escalated to an incident, now resolved: https://status.balena.io/incidents/yfyfmc0dxw52

@wwalker, @daghemo, I wanted to also share an alternative to the ARM cloud builders or to emulators, which is to build on a local ARM device like a Raspberry Pi 3B (running a recent version of balenaOS). Enable local mode on the device, then run the commands below on a terminal on your laptop/desktop, replacing ‘192.168.2.3’ with your local device’s IP address:

# build the docker image on the device at the given IP address
balena build -a my_app -h 192.168.2.3 -p 2375 .
# copy the docker image from the device to the laptop/desktop
ssh root@192.168.2.3 -p 22222 balena-engine save my_image | docker load
# upload the image to the balena cloud for deployment to devices
balena deploy my_app my_image

The documentation for the build and deploy commands can be read with balena help build and balena help deploy, or on this page: https://www.balena.io/docs/reference/cli/#build-source-

The ssh command line above copies the image to the laptop’s / deskotp’s local docker daemon, which therefore must be installed and running. The images can be listed with the docker ps command. The counterpart command to copy an image in the opposite direction (from the laptop/desktop to a balena device) is:

docker save my_image | ssh root@192.168.2.3 -p 22222 balena-engine load

daghemo · February 12, 2019, 10:26pm

@pdcastro, I’m going to try the suggested setup in the next few days.
Maybe arm01 is still having some problems:

[main]     E: Failed to fetch http://deb.debian.org/debian/dists/stretch/main/binary-armhf/Packages  Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device)
[main]     E: Failed to fetch http://deb.debian.org/debian/dists/stretch-updates/main/binary-armhf/Packages  Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device)
[main]     E: Failed to fetch http://security.debian.org/debian-security/dists/stretch/updates/main/binary-armhf/Packages  Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device)
[main]     E: Failed to fetch http://archive.raspberrypi.org/debian/dists/stretch/main/binary-armhf/Packages  Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device) [IP: 93.93.128.230 80]
[main]     E: Failed to fetch http://archive.raspbian.org/raspbian/dists/stretch/contrib/binary-armhf/Packages  Error writing to output file - write (28: No space left on device) Error writing to file - write (28: No space left on device)
[main]     E: Some index files failed to download. They have been ignored, or old ones used instead.

pdcastro · February 12, 2019, 11:21pm

@daghemo, we haven’t been able to reproduce the No space left on device errors, so I think it was a temporary issue. I have updated one of my posts above to list git and balena commands to force an emulated cloud build on x86 servers instead of using the native ARM servers, if you find that they are getting in the way rather than speeding up the build:

brdrcol · February 13, 2019, 4:09am

@pdcastro, our applications have also been getting the No space left on device errors on arm01 today, 4 hours or so after your reply

Thanks those commands though, I’m running a balena-emulated branch build now. We were previously using the balena-nocache remote branch but things seem to be building OK so far.

daghemo · February 13, 2019, 7:24am

Thanks @pdcastro!
Now I’m having a new error from arm01:

[main]     Step 25/71 : ENV UDEV on
[main]     Using cache
[main]      ---> 9cf325d34ba8
[main]     Step 26/71 : ENV INITSYSTEM on
[main]     Using cache
[main]      ---> d434e5ff1fbd
[main]     Step 27/71 : ENV container docker
[main]     unable to find image "sha256:1937a5dd9d24b82834ac1a3143369a0247bcd4f6b3c82159ee3783e77e7f4468"
[Info]     Uploading images
[Success]  Successfully uploaded images
[

cyplo · February 13, 2019, 10:09am

For the no space left error - there was an issue with one of our ARM builders - see https://status.balena.io/incidents/hjhvb0cf8pm7 and https://status.balena.io/incidents/yfyfmc0dxw52 - this should be resolved now.

pdcastro · February 13, 2019, 2:40pm

@daghemo, if you’re still seeing the ‘unable to find image’ error, try bypassing the builder cache:

git push balena master:balena-nocache
balena push ... --nocache
balena build ... --nocache
balena deploy ... --nocache

wwalker · February 14, 2019, 12:27am

No longer experiencing issues.

yyogeshwar · April 13, 2020, 6:30pm

Today i am also facing similar issue

Device - jetson nano ,

Tried 3-4 times building but same issue for around 30-40 minutes.

OS is production so can not use local mode.

anujdeshpande · April 14, 2020, 8:20am

Hi
I can confirm that at the moment we aren’t having any issues on our builders (https://status.balena.io/)
Are you still facing issues? Do you have complete logs of when you were trying to deploy?

Topic		Replies	Views
balena push builds failing to pull previous images Product support builder , docker , balena-cli	5	275	December 15, 2023
Problem building - read ECONNRESET Product support	24	3420	August 13, 2019
Native arm builders Product support	2	199	December 10, 2022
Native ARM builders [beta] Product support	3	2046	November 27, 2015
Many failed builds over the last 24 hours Product support builder	1	236	July 27, 2023

Experiencing Slow Build

Related topics