Problem building - read ECONNRESET

enricopenzo · August 12, 2019, 4:22pm

Hi,
I deployed several times today my project using

git push balena master

However, at some point I suddenly got the following error:

I was reading from your status page that this morning you deployed a fix for a problem related to ARM3:

I noticed that this morning my succesull builds started with arm01

while everytime I try to deploy now I get "building on arm03

Can this be a issue related to your new fix?

thundron · August 12, 2019, 4:53pm

Hi, it is due to an incident we had some hours ago, but it should be solved by now. We’ll investigate and let you know

Tschebbischeff · August 12, 2019, 4:54pm

Repost from here: An error occured: read ECONNRESET - airmonitor build

It doesn’t seem like the problem is fixed as is written on the status page. Just wanted to make sure you are aware of it not being fixed.

Greetings and thanks for your efforts

Edit: In the meantime for everybody having this problem, you can enforce a build that is locally emulated (i.e. definitely not running on arm03) as a workaround by running:

git push balena your-branch-name:balena-emulated

or if you are using the balena CLI to push your code:

balena push yourApplication --emulated

(I think this disables caching though )

enricopenzo · August 12, 2019, 5:38pm

Thanks for the tip.
I tried with your solution but unfortunately I’m having issues installing node

I will wait a bit and see if the issue with the builder will be resolved by the balena team

thundron · August 12, 2019, 5:42pm

@enricopenzo @tschebbischeff Could you please give it another try now with the standard builders (i.e. not emulated)?

enricopenzo · August 12, 2019, 6:37pm

Just tried but I got the same result

Craigson · August 12, 2019, 6:59pm

I’m getting the same error with a build system that’s unchanged and hasn’t had issues previously.

[debug] handling message: {"message":"\u001b[34m[logspout]\u001b[39m       (22/22) Installing mercurial (4.9.1-r0)"}
[logspout]       (22/22) Installing mercurial (4.9.1-r0)
[debug] handling message: {"message":"\u001b[34m[logspout]\u001b[39m       Executing busybox-1.30.1-r2.trigger"}
[logspout]       Executing busybox-1.30.1-r2.trigger
[debug] handling message: {"message":"\u001b[34m[logspout]\u001b[39m       OK: 456 MiB in 91 packages"}
[logspout]       OK: 456 MiB in 91 packages
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Uploading images"}
[Info]           Uploading images
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[31m[Error]\u001b[39m          An error occured: read ECONNRESET","isError":true}
[Error]          An error occured: read ECONNRESET
[debug] handling message: {"message":"\u001b[31m[Error]\u001b[39m          Not deploying release.","isError":true}
[Error]          Not deploying release.
RemoteBuildFailedError: Remote build failed
    at Bluebird.then (/usr/local/lib/node_modules/balena-cli/build/utils/remote-build.js:53:19)
    at tryCatcher (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/promise.js:517:31)
    at Promise._settlePromise (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/promise.js:574:18)
    at Promise._settlePromise0 (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/promise.js:619:10)
    at Promise._settlePromises (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/promise.js:699:18)
    at _drainQueueStep (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/async.js:138:12)
    at _drainQueue (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/async.js:131:9)
    at Async._drainQueues (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/async.js:147:5)
    at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
    at process.topLevelDomainCallback (domain.js:120:23)

If you need help, don't hesitate in contacting our support forums at
https://forums.balena.io

For bug reports or feature requests, have a look at the GitHub issues or
create a new one at: https://github.com/balena-io/balena-cli/issues/

pdcastro · August 12, 2019, 7:05pm

Thank you for these reports. I have not been able to reproduce the issue myself, but I see some ECONNRESET in builder logs and I have asked our dev ops team to investigate.

Craigson · August 12, 2019, 7:10pm

I can confirm it’s not working with and without the --emulated flag

[Info]           Dashboard link: https://dashboard.balena-cloud.com/apps/1477582/devices
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Building on arm03"}
[Info]           Building on arm03
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Pulling previous images for caching purposes..."}
[Info]           Pulling previous images for caching purposes...

pdcastro · August 12, 2019, 7:37pm

The dev ops team has been investigating this issue. It seems to affect only a subset of ARM builds, for some reason – potentially a specific ARM instruction set, or 32 vs 64 bits. We’ll update it here when we know more. Thanks for the reports.

cnr · August 12, 2019, 7:57pm

Also affecting my builds (both pushes to a raspberry pi and to an nvidia Tx2). Thank you for the reports

Would it be useful if I put together a miminal Dockerfile that replicates the result? Or is the devps team good to go?

pdcastro · August 12, 2019, 8:26pm

@cnr, if you have something to share, it will be welcome. We’ve got examples of Dockerfiles that fail with base images like “FROM balenalib/raspberrypi3”, and Dockerfiles that succeed with base images like “FROM arm32v7/busybox”, but the key difference it’s not yet clear – it may not even be the base image, but rather the application type, or a combination of them.

Craigson · August 12, 2019, 8:34pm

It doesn’t seem to be consistent, at least from my side. I’ve had one deployment succeed, which was simply after several retries. All on RPi3.

chrisys · August 12, 2019, 8:37pm

The inconsistencies may be a result of our team working on it - we’re still continuing to investigate the issue.

etc3 · August 12, 2019, 9:46pm

Also having the same issue today, with stable project with only minor source changes. Build fails as shown below:

[wificonnect]  Step 3/7 : RUN install_packages dnsmasq wireless-tools
[wificonnect]   ---> Running in f2dc5ab0c4f4
[pigpio]       gcc -O3 -Wall -pthread -fpic -c -o pigpio.o pigpio.c
[Info]         Uploading images
[Error]        An error occured: read ECONNRESET
[Error]        Not deploying release.

remote: error: hook declined to update refs/heads/master

Craigson · August 12, 2019, 10:14pm

Hi @chrisys, I understand your team is hard at work on solving the problem - but is there any update on what exactly the issue is or when it might be resolved? I have a project going live in two days and right now I can’t test/deploy to any of my devices…

pdcastro · August 12, 2019, 10:15pm

Update: a native ARM builder (arm03) has been taken out of operation while the problem is investigated, being replaced with QEMU emulation on x86 servers. This may cause ARM user builds to run more slowly, but a larger fraction of builds should be successful. (The only builds that might still fail are builds that are not compatible with QEMU emulation, which are relatively rare.)
A new incident was also created on the status page: https://status.balena.io/incidents/zq5zcjnr2v6l

pdcastro · August 12, 2019, 10:22pm

@Craigson, are you still getting ECONNRESET errors specifically? Because the temporary QEMU emulation on x86 servers (now in place) was supposed to have avoided those errors. If you’re getting a different error, we can investigate it too.

Craigson · August 12, 2019, 10:23pm

Thanks @pdcastro, right now I’m waiting 20+ mins for multi container projects to build, after which they seem to stall. I will restart a build now and report back asap.

[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m           Still Working..."}
[Info]           Still Working...

BryanMontes · August 12, 2019, 11:30pm

Hello guys, hope everyone has a nice weekend.
I am having troubles trying to deploy my Node application to a Balena Board. I got the same error message as enricopenzo.

I am trying to push a version that must be reviewed by the QA team, but in fact, it wasn’t possible because of the ECONNRESET.

I tried many times using the following:

git push myBalenaBoard <my-branch>:master
git push myBalenaBoard <my-branch>:balena-emulated
git push myBalenaBoard <my-branch>:balena-nocache
git push myBalenaBoard <my-branch>:refs/heads/balena-nocache

None of them worked.
The last build I ran took 1 hour 36 minutes to complete and at the end, it failed.

I did the same steps removing my package-lock.json files and still got the same error over and over again.
Last Friday, I was able to push a couple of builds with success using the same packages, nothing changed at all, only a few bugs solved on code, nothing else than that.

I get 2 kinds of errors:
1.- The build gets stuck installing npm libraries, and suddenly breaks. (This only happens when using “balena-emulated”)
2.- The build crashes when trying to get libraries from a repo and throws An error occurred: read ECONNRESET (this happens when pushing to master or balena-nocache)

Details:
Using ARM arch

Hope you guys find the root cause, if you need more information please let me know and I will be glad to give you information from the logs I got.

Thanks,
Bryan

Topic		Replies	Views
An error occured: read ECONNRESET - airmonitor build Product support	9	1203	August 14, 2019
Experiencing Slow Build Product support	15	1809	April 14, 2020
Failed to build (arm01) Product support	10	467	July 21, 2020
BalenaSense - error on build/deploy Product support	4	845	May 11, 2019
Infinite Building/Updating RPi3 Product support	17	446	January 31, 2020

Problem building - read ECONNRESET

Related topics