As another data point, it’s not always ECONNRESET: I just pushed my Dockerfile to the Xavier and, after multiple attempts at connecting (see above), it actually built successfully! (The overall build took several hours – no error here! Plus, ping balena.local was working fine throughout the build process, too.) But when it was time for balena-cli to connect to the supervisor’s REST API again, the same bug surfaced once more:
[Build] [feedback_screen] Successfully built 3f37eb21908b
[Build] [feedback_screen] Successfully tagged local_image_my_app:latest
[Debug] Setting device state...
[Debug] Sending request to http://192.168.2.144:48484/v2/local/target-state
ServiceUnavailableAPIError
at Request._callback (/snapshot/versioned-source/build/utils/device/api.js:147:39)
at Request.self.callback (/snapshot/versioned-source/node_modules/request/request.js:185:22)
at Request.emit (events.js:315:20)
at Request.EventEmitter.emit (domain.js:482:12)
at Request.<anonymous> (/snapshot/versioned-source/node_modules/request/request.js:1154:10)
at Request.emit (events.js:315:20)
at Request.EventEmitter.emit (domain.js:482:12)
at IncomingMessage.<anonymous> (/snapshot/versioned-source/node_modules/request/request.js:1076:12)
at Object.onceWrapper (events.js:421:28)
at IncomingMessage.emit (events.js:327:22)
at IncomingMessage.EventEmitter.emit (domain.js:482:12)
at endReadableNT (_stream_readable.js:1221:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21)
I also ran curl localhost:48484 on the device multiple times while the build process was going on and in all cases the HTTP request timed out.
–
Meanwhile, my colleague also gave it another try on his Xavier. He’s seeing the exact same logs from the supervisor as I am and the exact same behavior of curl localhost:48484 on the device. When he tries to push balena-os/wifi-connect, he, too, gets varying responses:
$ balena push balena.local --debug
[debug] new argv=[/usr/local/lib/balena-cli/bin/node,/usr/local/lib/balena-cli/bin/run,push,balena.local] length=4
[Debug] Using build source directory: .
[Debug] Pushing to local device: balena.local
[Debug] Checking we can access device
[Debug] Sending request to http://192.168.2.118:48484/ping
[Debug] Checking device supervisor version: 12.3.5
[Info] Starting build on device 192.168.2.118
[Debug] Loading project...
[Debug] Resolving project...
[Debug] docker-compose.yml file found at "."
[Debug] Creating project...
[Debug] Tarring all non-ignored files...
[Debug] Sending request to http://192.168.2.118:48484/v2/local/device-info
[Debug] Found build tasks:
[Debug] wifi-connect: image pull [balena/wifi-connect:aarch64]
[Debug] Resolving services with [jetson-nano|aarch64]
[Debug] Found project types:
[Debug] wifi-connect: External image
[Debug] Probing remote daemon for cache images
[Debug] Using 7 on-device images for cache...
[Debug] Starting builds...
[Debug] Setting device state...
[Debug] Sending request to http://192.168.2.118:48484/v2/local/target-state
[Debug] Sending target state: {"local":{"name":"local","config":{"SUPERVISOR_POLL_INTERVAL":"60000","SUPERVISOR_INSTANT_UPDATE_TRIGGER":"true","SUPERVISOR_LOCAL_MODE":"true","SUPERVISOR_CONNECTIVITY_CHECK":"true","SUPERVISOR_LOG_CONTROL":"true","SUPERVISOR_DELTA":"false","SUPERVISOR_DELTA_REQUEST_TIMEOUT":"30000","SUPERVISOR_DELTA_APPLY_TIMEOUT":"0","SUPERVISOR_DELTA_RETRY_COUNT":"30","SUPERVISOR_DELTA_RETRY_INTERVAL":"10000","SUPERVISOR_DELTA_VERSION":"2","SUPERVISOR_OVERRIDE_LOCK":"false","SUPERVISOR_PERSISTENT_LOGGING":"false","HOST_FIREWALL_MODE":"","HOST_DISCOVERABILITY":"true","SUPERVISOR_VPN_CONTROL":"false"},"apps":{"1":{"name":"localapp","commit":"localrelease","releaseId":"1","services":{"1":{"environment":{},"labels":{"io.balena.features.dbus":"1","io.balena.features.firmware":"1"},"restart":"always","network_mode":"host","privileged":true,"imageId":1,"serviceName":"wifi-connect","serviceId":1,"image":"local_image_wifi-connect:latest","running":true}},"volumes":{},"networks":{}}}},"dependent":{"apps":[],"devices":[]}}
[Debug] Sending request to http://192.168.2.118:48484/v2/local/target-state
[Info] Streaming device logs...
[Debug] Sending request to http://192.168.2.118:48484/v2/local/device-info
[Live] Watching for file changes...
[Live] Waiting for device state to settle...
[Debug] Device state not settled, retrying in 1000ms
[Logs] [4/13/2021, 5:45:25 PM] Installing service 'wifi-connect sha256:1d7bdc214cd9351a326e440229aee10f73dcb0778b5be4c9e17f6388d94d6002'
[Logs] [4/13/2021, 5:45:25 PM] Installed service 'wifi-connect sha256:1d7bdc214cd9351a326e440229aee10f73dcb0778b5be4c9e17f6388d94d6002'
[Logs] [4/13/2021, 5:45:25 PM] Starting service 'wifi-connect sha256:1d7bdc214cd9351a326e440229aee10f73dcb0778b5be4c9e17f6388d94d6002'
[Logs] [4/13/2021, 5:45:26 PM] Started service 'wifi-connect sha256:1d7bdc214cd9351a326e440229aee10f73dcb0778b5be4c9e17f6388d94d6002'
[Debug] Device state not settled, retrying in 1000ms
[Live] Device state settled
[Logs] [4/13/2021, 5:45:31 PM] [wifi-connect] Checking internet connectivity ...
[Logs] [4/13/2021, 5:45:32 PM] [wifi-connect] Your device is already connected to the internet.
[Logs] [4/13/2021, 5:45:32 PM] [wifi-connect] Skipping setting up Wifi-Connect Access Point. Will check again in 120 seconds
[Logs] [4/13/2021, 5:47:32 PM] [wifi-connect] Checking internet connectivity ...
[Logs] [4/13/2021, 5:47:33 PM] [wifi-connect] Your device is already connected to the internet.
[Logs] [4/13/2021, 5:47:33 PM] [wifi-connect] Skipping setting up Wifi-Connect Access Point. Will check again in 120 seconds
[Warn] Connection to device lost
Retrying "Streaming logs" after 1.0s (1 of 1000) due to: DeviceConnectionLostError: Connection to device lost
Retrying "Streaming logs" after 2.0s (2 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (3 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (4 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (5 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (6 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (7 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (8 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (9 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (10 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (11 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (12 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (13 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (14 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (15 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (16 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (17 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (18 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (19 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (20 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (21 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (22 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (23 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (24 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (25 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (26 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (27 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (28 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
[Live] Cleaning up device...
[Debug] Cleaning up done.
Task aborted on SIGINT signal
SIGINTError: Task aborted on SIGINT signal
at process.sigintHandler (/usr/local/lib/balena-cli/build/utils/helpers.js:315:20)
at Object.onceWrapper (events.js:421:26)
at process.emit (events.js:314:20)
at process.EventEmitter.emit (domain.js:483:12)
$ balena push balena.local --debug
[debug] new argv=[/usr/local/lib/balena-cli/bin/node,/usr/local/lib/balena-cli/bin/run,push,balena.local] length=4
[Debug] Using build source directory: .
[Debug] Pushing to local device: balena.local
[Debug] Checking we can access device
[Debug] Sending request to http://192.168.2.120:48484/ping
ECONNRESET: socket hang up
Error: socket hang up
at connResetException (internal/errors.js:609:14)
at Socket.socketOnEnd (_http_client.js:458:23)
at Socket.emit (events.js:326:22)
at Socket.EventEmitter.emit (domain.js:483:12)
at endReadableNT (_stream_readable.js:1241:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21)
For further help or support, visit:
https://www.balena.io/docs/reference/balena-cli/#support-faq-and-troubleshooting
(In fact, now that I’m thinking about it: Would a supervisor upgrade via balenaCloud even work if the supervisor’s API is unreachable?)