Problems doing a 'balena push' (free tier)

General
1

I’m using the Balena free tier to develop an app which uses a RPi to present a web UI for sending commands over USB to modelling guitar amplifiers from the Fender Mustang LT series (project outline page is work in progress but visible at https://docs.google.com/document/d/16ZGsf0yGLheq5xF0jEiFNVfTS9s3KD7a81lxxmo3TtA/edit?usp=sharing).

For the last couple of weeks I’ve had some problems running the ‘balena push’ command - to send updates of the project for building - its been failing about 30-50% of the time, but usually works on a second or third attempt, especially if I add –debug to the command line.

Today, since about for about an hour before sending this post, it just wasn’t working at all for about an hour (just came back as I finished writing this up).

With the –debug command line, the balena CLI output typically looks like this:

balena push --debug --draft --source deployment/balena fhau-staging
[debug] new argv=[/home/tim/Applications/balena/bin/node,/home/tim/Applications/balena/bin/run,push,–draft,–source,deployment/balena,fhau-staging] length=7
[debug] Deprecation check: 6.08736 days since last npm registry query for next major version release date.
[debug] Will not query the registry again until at least 7 days have passed.
[Debug] Using build source directory: deployment/balena
[Debug] Pushing to cloud for fleet: fhau-staging
[debug] Event tracking error: Timeout awaiting ‘response’ for 0ms
| Packaging the project source…[Debug] Tarring all non-ignored files…
/ Packaging the project source…[Debug] docker-compose.yml file found at “/home/tim/github/maneline/deployment/balena”
Packaging the project source…[Debug] Tarring complete in 221 ms
[debug] Connecting to builder at https://builder.balena-cloud.com/v3/build?slug=gh_tim_littlefair%2Ffhau-staging&dockerfilePath=&emulated=false&nocache=false&headless=false&isdraft=true
Oops something went wrong, please check your connection and try again.

RequestError
at ClientRequest. (/home/tim/Applications/balena/node_modules/got/dist/source/core/index.js:970:111)
at Object.onceWrapper (node:events:633:26)
at ClientRequest.emit (node:events:530:35)
at origin.emit (/home/tim/Applications/balena/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)
at emitErrorEvent (node:_http_client:104:11)
at TLSSocket.socketErrorListener (node:_http_client:518:5)
at TLSSocket.emit (node:events:518:28)
at emitErrorNT (node:internal/streams/destroy:170:8)
at emitErrorCloseNT (node:internal/streams/destroy:129:3)
at process.processTicksAndRejections (node:internal/process/task_queues:90:21)AggregateError [ETIMEDOUT]:
at internalConnectMultiple (node:net:1139:18)
at internalConnectMultiple (node:net:1215:5)
at Timeout.internalConnectMultipleTimeout (node:net:1741:5)
at listOnTimeout (node:internal/timers:590:11)
at process.processTimers (node:internal/timers:523:7)

I am conscious that I’ve been using Balena’s services fairly intensively (3-20 pushes per day over the recent period), and as a free tier user working on an hobby-related app which is never going to give rise to a fleet of tens or hundreds of devices or evolve into a product/service for which Balena’s paid tiers would be relevant, I understand that there may be some degree of throttling in place to prevent excessive use, but if my experience is due to CPU cycle rationing it would be good to have a hint of how long I should pause for before trying again.

I’ve enabled support access to the fleet in case anyone within Balena wants to look at it, but for now it seems to be working again.

2

Hello @tim-littlefair,

We opened Incident: Builder partially degraded service. It’s current status is Monitoring: A fix has been implemented and we are monitoring the results. So it is not fully resolved. But it would be helpful to know - are you still experiencing the issue?

3

Thanks for looking at this.
I’ve broken my deploy script at the moment, but once I’ve fixed it I’ll
do a test push, and a couple more later in the (Australian, UTC+8) day.

I’ll let you know how I go.

4

My first attempt after (hopefully) fixing the deploy script failed a few seconds ago like the ones I’ve had over the last week or two:

balena push --debug --draft --source _work/balena_fakeroot-0.2.0+beta0026.33f6945.2 fhau-staging
[debug] new argv=[/home/tim/Applications/balena/bin/node,/home/tim/Applications/balena/bin/run,push,--draft,--source,_work/balena_fakeroot-0.2.0+beta0026.33f6945.2,fhau-staging] length=7
[debug] Deprecation check: 0.72311 days since last npm registry query for next major version release date.
[debug] Will not query the registry again until at least 7 days have passed.
[Debug]   Using build source directory: _work/balena_fakeroot-0.2.0+beta0026.33f6945.2 
[Debug]   Pushing to cloud for fleet: fhau-staging
[debug] Event tracking error: Timeout awaiting 'response' for 0ms
| Packaging the project source...[Debug]   Tarring all non-ignored files...
/ Packaging the project source...[Debug]   docker-compose.yml file found at "/home/tim/github/maneline/_work/balena_fakeroot-0.2.0+beta0026.33f6945.2"
- Packaging the project source...[Debug]   Tarring complete in 227 ms
[debug] Connecting to builder at https://builder.balena-cloud.com/v3/build?slug=gh_tim_littlefair%2Ffhau-staging&dockerfilePath=&emulated=false&nocache=false&headless=false&isdraft=true
Oops something went wrong, please check your connection and try again.

RequestError
    at ClientRequest.<anonymous> (/home/tim/Applications/balena/node_modules/got/dist/source/core/index.js:970:111)
    at Object.onceWrapper (node:events:633:26)
    at ClientRequest.emit (node:events:530:35)
    at origin.emit (/home/tim/Applications/balena/node_modules/@szmarczak/http-timer/dist/source/index.js:43:20)
    at emitErrorEvent (node:_http_client:104:11)
    at TLSSocket.socketErrorListener (node:_http_client:518:5)
    at TLSSocket.emit (node:events:518:28)
    at emitErrorNT (node:internal/streams/destroy:170:8)
    at emitErrorCloseNT (node:internal/streams/destroy:129:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:90:21)AggregateError [ETIMEDOUT]: 
    at internalConnectMultiple (node:net:1139:18)
    at internalConnectMultiple (node:net:1215:5)
    at Timeout.internalConnectMultipleTimeout (node:net:1741:5)
    at listOnTimeout (node:internal/timers:590:11)
    at process.processTimers (node:internal/timers:523:7)

For further help or support, visit:
https://www.balena.io/docs/reference/balena-cli/#support-faq-and-troubleshooting

It could be a problem with the state of the directory I’m deploying from - I’ve been modifying the script to gather subdirectories and files scattered across a repo where balena is sharing resources with other deployment styles and I don’t want to tar the whole thing, before yesterday it was using symlinks instead of gathering a disposable root. But the changes to the script are only yesterday and today and it has been working about 50% of the time over the last week or so.

5

4th attempt is working - this time I left the –debug flag off.

Failed due to an error on my part in the Dockerfile will fix that and try again.

6

After fixing the Dockerfile, the first attempt failed, then the second completed.

[maneline-web]  Successfully built 10563eee0d58
[Info]          Uploading images
[Success]       Successfully uploaded images
[Info]          Built on 4b2d7d0
[Success]       Release successfully created!
[Info]          Release: bc6d4646a39691d1568207ab1c2ff566 (id: 3669717)

The failed run doesn’t have any useful diagnostic as I didn’t have –debug turned on.

Anyway, thanks for looking at this, I’ll do a few more pushes later in the day (when the repo is in a better state than it is right now), and I’ll report back with a summary of successes and failures.

7

I should note that I’m running with outdated version 22.1.1 of the Balena CLI, so I’ll update that too before doing any more pushes.