I’m having problems again, and this time they seem to be occurring 100% of the time.
I’ve upgraded to to CLI 2.4.15, this didn’t fix the problem, but it did give me a debug stack trace which contained file/line number in the JS/node module source where the error was raised and handled - not sure if this change is intended or raises security concerns for Balena, but with the extra information I hacked in a small change to the Javascript to see if more detail helps.
The first push after I installed the new CLI returned this error trace:
balena push --debug --source _work/balena_fakeroot-0.2.0+beta0051.9646bea.2 fhau-staging
[debug] new argv=[/home/tim/github/ml-sdk-251022a/balena-22.4.15/bin/node,/home/tim/github/ml-sdk-251022a/balena-22.4.15/bin/run,push,--source,_work/balena_fakeroot-0.2.0+beta0051.9646bea.2,fhau-staging] length=6
[debug] Deprecation check: 6.24772 days since last npm registry query for next major version release date.
[debug] Will not query the registry again until at least 7 days have passed.
[Debug] Using build source directory: _work/balena_fakeroot-0.2.0+beta0051.9646bea.2
[Debug] Pushing to cloud for fleet: fhau-staging
[debug] Event tracking error: Timeout awaiting 'response' for 0ms
| Packaging the project source...[Debug] Tarring all non-ignored files...
/ Packaging the project source...[Debug] docker-compose.yml file found at "/home/tim/github/maneline/_work/balena_fakeroot-0.2.0+beta0051.9646bea.2"
| Packaging the project source...[Debug] Tarring complete in 452 ms
[debug] Connecting to builder at https://builder.balena-cloud.com/v3/build?slug=gh_tim_littlefair%2Ffhau-staging&dockerfilePath=&emulated=false&nocache=false&headless=false&isdraft=false
Oops something went wrong, please check your connection and try again.
RequestError
at ClientRequest.<anonymous> (/home/tim/github/ml-sdk-251022a/balena-22.4.15/node_modules/got/dist/source/core/index.js:970:111)
at Object.onceWrapper (node:events:634:26)
at ClientRequest.emit (node:events:531:35)
at origin.emit (/home/tim/github/ml-sdk-251022a/balena-22.4.15/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)
at emitErrorEvent (node:_http_client:107:11)
at TLSSocket.socketErrorListener (node:_http_client:574:5)
at TLSSocket.emit (node:events:519:28)
at emitErrorNT (node:internal/streams/destroy:170:8)
at emitErrorCloseNT (node:internal/streams/destroy:129:3)
at process.processTicksAndRejections (node:internal/process/task_queues:90:21)AggregateError [ETIMEDOUT]:
at internalConnectMultiple (node:net:1134:18)
at internalConnectMultiple (node:net:1210:5)
at Timeout.internalConnectMultipleTimeout (node:net:1742:5)
at listOnTimeout (node:internal/timers:590:11)
at process.processTimers (node:internal/timers:523:7)
I modified the file /home/tim/github/ml-sdk-251022a/balena-22.4.15/node_modules/@szmarczakszmarczakszmarczakszmarczakszmarczakszmarczakszmarczakszmarczak/http-timer/dist/source/index.js around line 43 so it looks like this:
const handleError = (origin) => {
const emit = origin.emit.bind(origin);
origin.emit = (event, ...args) => {
console.log("in origin.emit"); //TL
// Catches the `error` event
if (event === 'error') {
console.log("event==='error'"); //TL
timings.error = Date.now();
timings.phases.total = timings.error - timings.start;
origin.emit = emit;
console.log("timings:"); //TL
console.log(timings); //TL
console.log("origin:"); //TL
console.log(origin); //TL
}
console.log("returning 'emit'"); //TL
// Saves the original behavior
return emit(event, ...args);
};
};
With this version, I can see the content of the ‘timings’ and ‘origin’ objects. ‘origin’ contains about 1k lines of data including multiple mentions of authorization data from my balena CLI login session so I’ve redacted it but I’m happy to email a copy to anyone at Balena who wants to see it. The redacted output with the hacked javascript looks like this:
tim@tim-OptiPlex-7450-AIO:~/github/maneline$ time scripts/build_release.sh --deploy-balena-beta --remove-stale-fakeroot --fhau-staging --debug
buildString=0.2.0+beta0051.9646bea.2
buildGitRef=#9646bea + changes to: build_release.sh rebuild_sdk.sh
buildAndroidVersionCode=2000051
Balena already logged in
Removing stale directory _work/balena_fakeroot-0.2.0+beta0051.9646bea.2
Stale directory removed
Building deployment root at _work/balena_fakeroot-0.2.0+beta0051.9646bea.2
Files copied to _work/balena_fakeroot-0.2.0+beta0051.9646bea.2
Remaining arguments: --fhau-staging --debug
Will deploy to fhau-staging
balena push --debug --source _work/balena_fakeroot-0.2.0+beta0051.9646bea.2 fhau-staging
[debug] new argv=[/home/tim/github/ml-sdk-251022a/balena-22.4.15/bin/node,/home/tim/github/ml-sdk-251022a/balena-22.4.15/bin/run,push,--source,_work/balena_fakeroot-0.2.0+beta0051.9646bea.2,fhau-staging] length=6
[debug] Deprecation check: 6.27001 days since last npm registry query for next major version release date.
[debug] Will not query the registry again until at least 7 days have passed.
[Debug] Using build source directory: _work/balena_fakeroot-0.2.0+beta0051.9646bea.2
in origin.emit
returning 'emit'
in origin.emit
returning 'emit'
[Debug] Pushing to cloud for fleet: fhau-staging
in origin.emit
returning 'emit'
in origin.emit
returning 'emit'
in origin.emit
event==='error'
timings:
{
start: 1761120149057,
socket: 1761120149060,
lookup: 1761120149090,
connect: 1761120149098,
secureConnect: 1761120149109,
upload: 1761120149110,
response: undefined,
end: undefined,
error: 1761120149114,
abort: undefined,
phases: {
wait: 3,
dns: 30,
tcp: 8,
tls: 11,
request: 1,
firstByte: undefined,
download: undefined,
total: 57
}
}
origin:
** redacted **
returning 'emit'
Oops something went wrong, please check your connection and try again.
RequestError
at ClientRequest.<anonymous> (/home/tim/github/ml-sdk-251022a/balena-22.4.15/node_modules/got/dist/source/core/index.js:970:111)
at Object.onceWrapper (node:events:634:26)
at ClientRequest.emit (node:events:531:35)
at origin.emit (/home/tim/github/ml-sdk-251022a/balena-22.4.15/node_modules/@szmarczak/http-timer/dist/source/index.js:50:20)
at emitErrorEvent (node:_http_client:107:11)
at TLSSocket.socketErrorListener (node:_http_client:574:5)
at TLSSocket.emit (node:events:519:28)
at emitErrorNT (node:internal/streams/destroy:170:8)
at emitErrorCloseNT (node:internal/streams/destroy:129:3)
at process.processTicksAndRejections (node:internal/process/task_queues:90:21)AggregateError [ETIMEDOUT]:
at internalConnectMultiple (node:net:1134:18)
at internalConnectMultiple (node:net:1210:5)
at Timeout.internalConnectMultipleTimeout (node:net:1742:5)
at listOnTimeout (node:internal/timers:590:11)
at process.processTimers (node:internal/timers:523:7)
For further help or support, visit:
https://www.balena.io/docs/reference/balena-cli/#support-faq-and-troubleshooting
If I comment out the line which dumps the ‘origin’ object, it looks like there are two separate errors being handled with different combinations of populated items in the ‘timings’ object
buildString=0.2.0+beta0051.9646bea.2
buildGitRef=#9646bea + changes to: build_release.sh rebuild_sdk.sh
buildAndroidVersionCode=2000051
Balena already logged in
Removing stale directory _work/balena_fakeroot-0.2.0+beta0051.9646bea.2
Stale directory removed
Building deployment root at _work/balena_fakeroot-0.2.0+beta0051.9646bea.2
Files copied to _work/balena_fakeroot-0.2.0+beta0051.9646bea.2
Remaining arguments: --fhau-staging --debug
Will deploy to fhau-staging
balena push --debug --source _work/balena_fakeroot-0.2.0+beta0051.9646bea.2 fhau-staging
[debug] new argv=[/home/tim/github/ml-sdk-251022a/balena-22.4.15/bin/node,/home/tim/github/ml-sdk-251022a/balena-22.4.15/bin/run,push,--source,_work/balena_fakeroot-0.2.0+beta0051.9646bea.2,fhau-staging] length=6
[debug] Deprecation check: 6.28456 days since last npm registry query for next major version release date.
[debug] Will not query the registry again until at least 7 days have passed.
[Debug] Using build source directory: _work/balena_fakeroot-0.2.0+beta0051.9646bea.2
in origin.emit
returning 'emit'
in origin.emit
returning 'emit'
[Debug] Pushing to cloud for fleet: fhau-staging
in origin.emit
returning 'emit'
in origin.emit
returning 'emit'
in origin.emit
event==='error'
timings:
{
start: 1761121406140,
socket: 1761121406143,
lookup: 1761121406166,
connect: 1761121406173,
secureConnect: 1761121406186,
upload: 1761121406187,
response: undefined,
end: undefined,
error: 1761121406191,
abort: undefined,
phases: {
wait: 3,
dns: 23,
tcp: 7,
tls: 13,
request: 1,
firstByte: undefined,
download: undefined,
total: 51
}
}
origin:
returning 'emit'
[debug] Event tracking error: Timeout awaiting 'response' for 0ms
| Packaging the project source...[Debug] Tarring all non-ignored files...
/ Packaging the project source...[Debug] docker-compose.yml file found at "/home/tim/github/maneline/_work/balena_fakeroot-0.2.0+beta0051.9646bea.2"
| Packaging the project source...[Debug] Tarring complete in 465 ms
[debug] Connecting to builder at https://builder.balena-cloud.com/v3/build?slug=gh_tim_littlefair%2Ffhau-staging&dockerfilePath=&emulated=false&nocache=false&headless=false&isdraft=false
in origin.emit
returning 'emit'
/ Uploading source package to https://builder.balena-cloud.comin origin.emit
event==='error'
timings:
{
start: 1761121407552,
socket: 1761121407554,
lookup: 1761121407623,
connect: undefined,
secureConnect: undefined,
upload: undefined,
response: undefined,
end: undefined,
error: 1761121408128,
abort: undefined,
phases: {
wait: 2,
dns: 69,
tcp: undefined,
tls: undefined,
request: undefined,
firstByte: undefined,
download: undefined,
total: 576
}
}
origin:
returning 'emit'
Oops something went wrong, please check your connection and try again.
RequestError
at ClientRequest.<anonymous> (/home/tim/github/ml-sdk-251022a/balena-22.4.15/node_modules/got/dist/source/core/index.js:970:111)
at Object.onceWrapper (node:events:634:26)
at ClientRequest.emit (node:events:531:35)
at origin.emit (/home/tim/github/ml-sdk-251022a/balena-22.4.15/node_modules/@szmarczak/http-timer/dist/source/index.js:50:20)
at emitErrorEvent (node:_http_client:107:11)
at TLSSocket.socketErrorListener (node:_http_client:574:5)
at TLSSocket.emit (node:events:519:28)
at emitErrorNT (node:internal/streams/destroy:170:8)
at emitErrorCloseNT (node:internal/streams/destroy:129:3)
at process.processTicksAndRejections (node:internal/process/task_queues:90:21)AggregateError [ETIMEDOUT]:
at internalConnectMultiple (node:net:1134:18)
at internalConnectMultiple (node:net:1210:5)
at Timeout.internalConnectMultipleTimeout (node:net:1742:5)
at listOnTimeout (node:internal/timers:590:11)
at process.processTimers (node:internal/timers:523:7)
For further help or support, visit:
https://www.balena.io/docs/reference/balena-cli/#support-faq-and-troubleshooting
real 0m3.555s
user 0m1.391s
sys 0m0.407s
tim@tim-OptiPlex-7450-AIO:~/github/maneline$
I’d be very grateful if someone could take a look at this, as presently I can’t push by project using the CLI at all (going to have a try using the git method when I finish this comment).
Should just comment that I am based in Perth, Western Australia, in case this is a resource/timing issue related to how much CPU capacity is provisioned in different cloud regions.
EDIT: The git push workflow won’t work for this project because I have build steps in my CI which need to run before the Dockerfile is processed,