Hi,
I deployed several times today my project using
git push balena master
However, at some point I suddenly got the following error:
I was reading from your status page that this morning you deployed a fix for a problem related to ARM3:
I noticed that this morning my succesull builds started with arm01
while everytime I try to deploy now I get "building on arm03
Can this be a issue related to your new fix?
Hi, it is due to an incident we had some hours ago, but it should be solved by now. We’ll investigate and let you know
Repost from here: An error occured: read ECONNRESET - airmonitor build
It doesn’t seem like the problem is fixed as is written on the status page. Just wanted to make sure you are aware of it not being fixed.
Greetings and thanks for your efforts 
Edit: In the meantime for everybody having this problem, you can enforce a build that is locally emulated (i.e. definitely not running on arm03) as a workaround by running:
git push balena your-branch-name:balena-emulated
or if you are using the balena CLI to push your code:
balena push yourApplication --emulated
(I think this disables caching though 
)
Thanks for the tip.
I tried with your solution but unfortunately I’m having issues installing node
I will wait a bit and see if the issue with the builder will be resolved by the balena team
@enricopenzo @tschebbischeff Could you please give it another try now with the standard builders (i.e. not emulated)?
I’m getting the same error with a build system that’s unchanged and hasn’t had issues previously.
[debug] handling message: {"message":"\u001b[34m[logspout]\u001b[39m (22/22) Installing mercurial (4.9.1-r0)"}
[logspout] (22/22) Installing mercurial (4.9.1-r0)
[debug] handling message: {"message":"\u001b[34m[logspout]\u001b[39m Executing busybox-1.30.1-r2.trigger"}
[logspout] Executing busybox-1.30.1-r2.trigger
[debug] handling message: {"message":"\u001b[34m[logspout]\u001b[39m OK: 456 MiB in 91 packages"}
[logspout] OK: 456 MiB in 91 packages
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Uploading images"}
[Info] Uploading images
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[31m[Error]\u001b[39m An error occured: read ECONNRESET","isError":true}
[Error] An error occured: read ECONNRESET
[debug] handling message: {"message":"\u001b[31m[Error]\u001b[39m Not deploying release.","isError":true}
[Error] Not deploying release.
RemoteBuildFailedError: Remote build failed
at Bluebird.then (/usr/local/lib/node_modules/balena-cli/build/utils/remote-build.js:53:19)
at tryCatcher (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/util.js:16:23)
at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/promise.js:517:31)
at Promise._settlePromise (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/promise.js:574:18)
at Promise._settlePromise0 (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/promise.js:619:10)
at Promise._settlePromises (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/promise.js:699:18)
at _drainQueueStep (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/async.js:138:12)
at _drainQueue (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/async.js:131:9)
at Async._drainQueues (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/async.js:147:5)
at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/node_modules/balena-cli/node_modules/bluebird/js/release/async.js:17:14)
at runCallback (timers.js:705:18)
at tryOnImmediate (timers.js:676:5)
at processImmediate (timers.js:658:5)
at process.topLevelDomainCallback (domain.js:120:23)
If you need help, don't hesitate in contacting our support forums at
https://forums.balena.io
For bug reports or feature requests, have a look at the GitHub issues or
create a new one at: https://github.com/balena-io/balena-cli/issues/Thank you for these reports. I have not been able to reproduce the issue myself, but I see some ECONNRESET in builder logs and I have asked our dev ops team to investigate.
I can confirm it’s not working with and without the --emulated flag
[Info] Dashboard link: https://dashboard.balena-cloud.com/apps/1477582/devices
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Building on arm03"}
[Info] Building on arm03
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Pulling previous images for caching purposes..."}
[Info] Pulling previous images for caching purposes...The dev ops team has been investigating this issue. It seems to affect only a subset of ARM builds, for some reason – potentially a specific ARM instruction set, or 32 vs 64 bits. We’ll update it here when we know more. Thanks for the reports.
Also affecting my builds (both pushes to a raspberry pi and to an nvidia Tx2). Thank you for the reports
Would it be useful if I put together a miminal Dockerfile that replicates the result? Or is the devps team good to go?
@cnr, if you have something to share, it will be welcome. We’ve got examples of Dockerfiles that fail with base images like “FROM balenalib/raspberrypi3”, and Dockerfiles that succeed with base images like “FROM arm32v7/busybox”, but the key difference it’s not yet clear – it may not even be the base image, but rather the application type, or a combination of them.
It doesn’t seem to be consistent, at least from my side. I’ve had one deployment succeed, which was simply after several retries. All on RPi3.
The inconsistencies may be a result of our team working on it - we’re still continuing to investigate the issue.
Also having the same issue today, with stable project with only minor source changes. Build fails as shown below:
[wificonnect] Step 3/7 : RUN install_packages dnsmasq wireless-tools [wificonnect] ---> Running in f2dc5ab0c4f4 [pigpio] gcc -O3 -Wall -pthread -fpic -c -o pigpio.o pigpio.c [Info] Uploading images [Error] An error occured: read ECONNRESET [Error] Not deploying release. remote: error: hook declined to update refs/heads/master
Hi @chrisys, I understand your team is hard at work on solving the problem - but is there any update on what exactly the issue is or when it might be resolved? I have a project going live in two days and right now I can’t test/deploy to any of my devices…
Update: a native ARM builder (arm03) has been taken out of operation while the problem is investigated, being replaced with QEMU emulation on x86 servers. This may cause ARM user builds to run more slowly, but a larger fraction of builds should be successful. (The only builds that might still fail are builds that are not compatible with QEMU emulation, which are relatively rare.)
A new incident was also created on the status page: https://status.balena.io/incidents/zq5zcjnr2v6l
@Craigson, are you still getting ECONNRESET errors specifically? Because the temporary QEMU emulation on x86 servers (now in place) was supposed to have avoided those errors. If you’re getting a different error, we can investigate it too.
Thanks @pdcastro, right now I’m waiting 20+ mins for multi container projects to build, after which they seem to stall. I will restart a build now and report back asap.
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...
[debug] handling message: {"message":"\u001b[36m[Info]\u001b[39m Still Working..."}
[Info] Still Working...Hello guys, hope everyone has a nice weekend.
I am having troubles trying to deploy my Node application to a Balena Board. I got the same error message as enricopenzo.
I am trying to push a version that must be reviewed by the QA team, but in fact, it wasn’t possible because of the ECONNRESET.
I tried many times using the following:
git push myBalenaBoard <my-branch>:master
git push myBalenaBoard <my-branch>:balena-emulated
git push myBalenaBoard <my-branch>:balena-nocache
git push myBalenaBoard <my-branch>:refs/heads/balena-nocache
None of them worked.
The last build I ran took 1 hour 36 minutes to complete and at the end, it failed.
I did the same steps removing my package-lock.json files and still got the same error over and over again.
Last Friday, I was able to push a couple of builds with success using the same packages, nothing changed at all, only a few bugs solved on code, nothing else than that.
I get 2 kinds of errors:
1.- The build gets stuck installing npm libraries, and suddenly breaks. (This only happens when using “balena-emulated”)
2.- The build crashes when trying to get libraries from a repo and throws An error occurred: read ECONNRESET (this happens when pushing to master or balena-nocache)
Details:
Using ARM arch
Hope you guys find the root cause, if you need more information please let me know and I will be glad to give you information from the logs I got.
Thanks,
Bryan
