balena push fails to work with Jetson Xavier NX

Hi, both my colleague and I are experiencing very strange issues with BalenaOS (version 2.67.3+rev5) on our respective Jetson Xavier NX (dev kit), whereas everything is working fine on the Jetson Nano (dev kit).

No matter what container I try to balena push to the Xavier, I always end up with

$ balena push balena.local
[Info]    Starting build on device 192.168.2.144

and the process keeps hanging until, eventually, I get an ECONNRESET error (see below). Debug output:

$ balena push balena.local --debug                     130 ↵
[debug] new argv=[/home/user/opt/balena-cli/balena,/snapshot/versioned-source/bin/balena,push,balena.local] length=4
[Debug]   Using build source directory: . 
[Debug]   Pushing to local device: balena.local
[Debug]   Checking we can access device
[Debug]   Sending request to http://192.168.2.144:48484/ping  <---- Here it's hanging for a long time
ECONNRESET: read ECONNRESET
Error: read ECONNRESET
    at TCP.onStreamRead (internal/stream_base_commons.js:205:27)
For further help or support, visit:
https://www.balena.io/docs/reference/balena-cli/#support-faq-and-troubleshooting

Meanwhile, balena ssh balena.local is working fine and I can access the device without any problems. (I tried curl localhost:48484/ping on the device itself but that would keep hanging, too.)

FWIW before trying out Balena, both my colleague and I had used an SD card with Nvidia’s JetPack 4.5 Ubuntu image on the Xavier. This was working fine but what I’m trying to get at is: Maybe JetPack 4.5 is causing similar trouble on the Xavier as on the Nano?

I would appreciate any help!

I stand corrected: Turns out it sometimes works with some images (or it works half-way) but sooner or later I still end up with an ECONNRESET:

[Build]   [frontend] Step 11/13 : CMD ["npm", "start"]
[Build]   [frontend]  ---> Running in 65427545be8e
[Build]   [frontend] Removing intermediate container 65427545be8e
[Build]   [frontend]  ---> 356fc48611fd
[Build]   [frontend] Step 12/13 : LABEL io.resin.local.image=1
[Build]   [frontend]  ---> Running in 9781fa32c94f
[Build]   [frontend] Removing intermediate container 9781fa32c94f
[Build]   [frontend]  ---> 8bf67135d529
[Build]   [frontend] Step 13/13 : LABEL io.resin.local.service=frontend
[Build]   [frontend]  ---> Running in c83270b85f80
[Build]   [frontend] Removing intermediate container c83270b85f80
[Build]   [frontend]  ---> e0e1a21ee8e9
[Build]   [frontend] Successfully built e0e1a21ee8e9
[Build]   [frontend] Successfully tagged local_image_frontend:latest

ECONNRESET: read ECONNRESET

(Here, I tried to push GitHub - balena-io-examples/multicontainer-getting-started: Get up and running quickly with a multicontainer setup on balena)

balena push balena.local --debug
[debug] new argv=[/home/user/opt/balena-cli/balena,/snapshot/versioned-source/bin/balena,push,balena.local] length=4
[Debug]   Using build source directory: . 
[Debug]   Pushing to local device: balena.local
[Debug]   Checking we can access device
[Debug]   Sending request to http://192.168.2.144:48484/ping
[Debug]   Checking device supervisor version: 12.3.0
[Info]    Starting build on device 192.168.2.144
[Debug]   Loading project...
[Debug]   Resolving project...
[Debug]   docker-compose.yml file found at "."
[Debug]   Creating project...
[Debug]   Tarring all non-ignored files...
[Debug]   Sending request to http://192.168.2.144:48484/v2/local/device-info
ECONNRESET: read ECONNRESET

Error: read ECONNRESET

(Here I tried to push my own balenalib/%%BALENA_MACHINE_NAME%%-ubuntu:bionic-based container.)

Hi there, two quick thoughts on this one…

First and foremost, Jetpack 4.5 may definitely be the problem here, as you mentioned and linked, we do not have any support in place for 4.5 on the Nano, and it won’t even boot. Thus, even if the Xavier does boot with it, I would absolutely expect errors along the way.

However, a second thought is whether that balena.local domain is having issues, and instead try balena push <insert-your-ip-address-here> --debug

See if that works any better, thanks!

First and foremost, Jetpack 4.5 may definitely be the problem here, as you mentioned and linked, we do not have any support in place for 4.5 on the Nano, and it won’t even boot. Thus, even if the Xavier does boot with it, I would absolutely expect errors along the way.

Counterpoint: Wasn’t that “simply” because JetPack 4.5 for the Nano had changed the Nano’s SPI flash? Moreover, it prevented the Nano from booting altogether. With the Xavier this seems rather unlikely to me, since the Xavier is still booting BalenaOS without problems and SSH is working fine.

However, a second thought is whether that balena.local domain is having issues, and instead try balena push <insert-your-ip-address-here> --debug

Note that, according to the logs I posted, balena push will resolve the hostname name to the IP address right in the beginning and then connect to that IP, so that it doesn’t matter whether I use balena.local or not. In fact, while balena push is hanging at Sending request to http://192.168.2.144:48484/ping, I can still ping balena.local without problems.

I’m really starting to think it’s the web server behind balena.local:48484/ping that’s at fault here. Namely, whenever balena push is hanging, curl localhost:48484/ping (executed on the device via SSH) is hanging, too. And whenever balena push seems to be working for a moment, curl localhost:48484/ping on the device is working, too. (It’s returning OK.)

The weird thing is: Whether or not curl localhost:48484/ping works doesn’t seem to be related to whether the rest of the device is actually working or not. Right now, for instance, balena push balena.local is running fine for a moment and balena-engine on the Xavier is building my Docker image without problems. But on the device itself curl localhost:48484/ping is again timing out, so I’m sure it’ll only be a matter of time until balena push will complain and abort with an error message again.

After some further investigation, I’m increasingly convinced that it’s the web server on balena.local:48484 (which, as it turns out, is the balena-supervisor v12.3.0) that’s causing the issue.

Please find the logs below. Each of the “blocks” (which I separated by blank lines) has shown up repeatedly (in fact: dozens of times if not thousands of times) over the last few minutes. I’m merely trying to convey the unique pieces here.

$ balena-engine attach --no-stdin resin_supervisor

[error]   Non-empty stderr stream from journalctl log fetching:  Considering root directory '/run/log/journal'.
[error]     Root directory /run/log/journal added.
[error]   Considering directory '/run/log/journal/5a0e071f60894216aedc88334fdad282'.
[error]   Directory /run/log/journal/5a0e071f60894216aedc88334fdad282 added.
[error]   Journal effective settings seal=no compress=no compress_threshold_bytes=8B
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-0000000000085125-0005bfd70b25cabd.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-0000000000083bca-0005bfd70b1c59a9.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000008266f-0005bfd709075db4.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-0000000000081667-0005bfd708ff88cc.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000008037f-0005bfd7054943b8.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000007edd3-0005bfd70534533b.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000007d90c-0005bfd700a0e96f.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000007c655-0005bfd70051a8e8.journal added.
[error]   Considering root directory '/var/log/journal'.
[error]   Journal filter: _SYSTEMD_UNIT=balena.service
[error]   Reiterating files to get inotify watches established.
[error]   Considering root directory '/run/log/journal'.
[error]   Considering directory '/run/log/journal/5a0e071f60894216aedc88334fdad282'.
[error]   Considering root directory '/var/log/journal'.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal removed.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal removed.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000008667e-0005bfd70cae1581.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000007c655-0005bfd70051a8e8.journal removed.
[error]   Failed to open journal file /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000007c655-0005bfd70051a8e8.journal: No such file or directory
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal removed.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-0000000000087beb-0005bfd70f69f507.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000007d90c-0005bfd700a0e96f.journal removed.
[error]   Failed to open journal file /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000007d90c-0005bfd700a0e96f.journal: No such file or directory
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal removed.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-00000000000890b4-0005bfd7119e62ed.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000007edd3-0005bfd70534533b.journal removed.
[error]   Failed to open journal file /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000007edd3-0005bfd70534533b.journal: No such file or directory
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal removed.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000008a67a-0005bfd711b245d7.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000008037f-0005bfd7054943b8.journal removed.
[error]   Failed to open journal file /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000008037f-0005bfd7054943b8.journal: No such file or directory
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal removed.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000008b962-0005bfd711bed89b.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-0000000000081667-0005bfd708ff88cc.journal removed.
[error]   Failed to open journal file /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-0000000000081667-0005bfd708ff88cc.journal: No such file or directory



[error]   Non-empty stderr stream from journalctl log fetching:  Attempt to move to uninitialized object: 151720
[error]     Skipping message we can't read: Bad message



[error]   Non-empty stderr stream from journalctl log fetching:  Considering root directory '/run/log/journal'.
[error]     Root directory /run/log/journal added.
[error]   Considering directory '/run/log/journal/5a0e071f60894216aedc88334fdad282'.
[error]   Directory /run/log/journal/5a0e071f60894216aedc88334fdad282 added.
[error]   Journal effective settings seal=no compress=no compress_threshold_bytes=8B
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-0000000000085125-0005bfd70b25cabd.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-0000000000083bca-0005bfd70b1c59a9.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000008266f-0005bfd709075db4.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-0000000000081667-0005bfd708ff88cc.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000008037f-0005bfd7054943b8.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000007edd3-0005bfd70534533b.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000007d90c-0005bfd700a0e96f.journal added.
[error]   File /run/log/journal/5a0e071f60894216aedc88334fdad282/system@31def1d7817b4970be30135815500235-000000000007c655-0005bfd70051a8e8.journal added.
[error]   Considering root directory '/var/log/journal'.
[error]   Journal filter: _SYSTEMD_UNIT=balena.service
[error]   Insufficient watch descriptors available. Reverting to -n.
[error]   Root directory /run/log/journal removed.
[error]   Directory /run/log/journal/5a0e071f60894216aedc88334fdad282 removed.
[error]   mmap cache statistics: 1000 hit, 9 miss



[error]   Failed to iterate through journal: Bad message
[error]   Directory /run/log/journal/5a0e071f60894216aedc88334fdad282 removed.
[error]   Root directory /run/log/journal removed.
[error]   mmap cache statistics: 53250 hit, 33 miss
[error]   
[error]   Non-empty stderr stream from journalctl log fetching:  Attempt to move to uninitialized object: 126136
[error]     Failed to iterate through journal: Bad message
[error]   Root directory /run/log/journal removed.
[error]   Directory /run/log/journal/5a0e071f60894216aedc88334fdad282 removed.
[error]   mmap cache statistics: 44035 hit, 16 miss
[error]   
[error]   Non-empty stderr stream from journalctl log fetching:  Attempt to move to uninitialized object: 126136
[error]     Failed to iterate through journal: Bad message
[error]   Root directory /run/log/journal removed.
[error]   Directory /run/log/journal/5a0e071f60894216aedc88334fdad282 removed.
[error]   mmap cache statistics: 57436 hit, 14 miss
[error]   
[error]   Non-empty stderr stream from journalctl log fetching:  Attempt to move to uninitialized object: 126136
[error]     Failed to iterate through journal: Bad message
[error]   Directory /run/log/journal/5a0e071f60894216aedc88334fdad282 removed.
[error]   Root directory /run/log/journal removed.
[error]   mmap cache statistics: 52803 hit, 14 miss
[error]   
[error]   Non-empty stderr stream from journalctl log fetching:  Attempt to move to uninitialized object: 126136
[error]     Failed to iterate through journal: Bad message
[error]   Directory /run/log/journal/5a0e071f60894216aedc88334fdad282 removed.
[error]   Root directory /run/log/journal removed.
[error]   mmap cache statistics: 46475 hit, 16 miss
[error]   
[error]   Non-empty stderr stream from journalctl log fetching:  Attempt to move to uninitialized object: 126136
[error]     Failed to iterate through journal: Bad message
[error]   Directory /run/log/journal/5a0e071f60894216aedc88334fdad282 removed.
[error]   Root directory /run/log/journal removed.
[error]   mmap cache statistics: 51785 hit, 16 miss



[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
[debug]   Spawning journald with: chroot  /mnt/root journalctl -a --follow -o json

The output of journalctl -au resin-supervisor is pretty much the same.

Since the image for Xavier NX is already comparatively old, I’ve had a look at the supervisor’s changelog – maybe this bug has already been fixed in newer versions?

I’m not at all sure whether the logs above have anything to do with our issue (I’m really fishing in muddy waters here) but the following change seems at least somewhat related:

# v12.4.3
## (2021-02-24)
* Refactor journalctl monitor to only spawn new process on exit [Miguel Casqueira]

Moreover, there’s also the following instance of “something getting stuck”:

# v12.3.7
## (2021-02-16)
* Patch killServicesUsingApi to not get stuck in noop loop [Miguel Casqueira]

Finally, there’s also this (open?) bug report which seems similar to our issue in that it also results in balena push throwing ECONNRESET.

Anyway, we would really appreciate your help here because I don’t think my colleague and I will get any further on our own.

As another data point, it’s not always ECONNRESET: I just pushed my Dockerfile to the Xavier and, after multiple attempts at connecting (see above), it actually built successfully! (The overall build took several hours – no error here! Plus, ping balena.local was working fine throughout the build process, too.) But when it was time for balena-cli to connect to the supervisor’s REST API again, the same bug surfaced once more:

[Build]   [feedback_screen] Successfully built 3f37eb21908b
[Build]   [feedback_screen] Successfully tagged local_image_my_app:latest
[Debug]   Setting device state...
[Debug]   Sending request to http://192.168.2.144:48484/v2/local/target-state

ServiceUnavailableAPIError
    at Request._callback (/snapshot/versioned-source/build/utils/device/api.js:147:39)
    at Request.self.callback (/snapshot/versioned-source/node_modules/request/request.js:185:22)
    at Request.emit (events.js:315:20)
    at Request.EventEmitter.emit (domain.js:482:12)
    at Request.<anonymous> (/snapshot/versioned-source/node_modules/request/request.js:1154:10)
    at Request.emit (events.js:315:20)
    at Request.EventEmitter.emit (domain.js:482:12)
    at IncomingMessage.<anonymous> (/snapshot/versioned-source/node_modules/request/request.js:1076:12)
    at Object.onceWrapper (events.js:421:28)
    at IncomingMessage.emit (events.js:327:22)
    at IncomingMessage.EventEmitter.emit (domain.js:482:12)
    at endReadableNT (_stream_readable.js:1221:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)

I also ran curl localhost:48484 on the device multiple times while the build process was going on and in all cases the HTTP request timed out.

Meanwhile, my colleague also gave it another try on his Xavier. He’s seeing the exact same logs from the supervisor as I am and the exact same behavior of curl localhost:48484 on the device. When he tries to push balena-os/wifi-connect, he, too, gets varying responses:

$ balena push balena.local --debug
[debug] new argv=[/usr/local/lib/balena-cli/bin/node,/usr/local/lib/balena-cli/bin/run,push,balena.local] length=4
[Debug]   Using build source directory: .
[Debug]   Pushing to local device: balena.local
[Debug]   Checking we can access device
[Debug]   Sending request to http://192.168.2.118:48484/ping
[Debug]   Checking device supervisor version: 12.3.5
[Info]    Starting build on device 192.168.2.118
[Debug]   Loading project...
[Debug]   Resolving project...
[Debug]   docker-compose.yml file found at "."
[Debug]   Creating project...
[Debug]   Tarring all non-ignored files...
[Debug]   Sending request to http://192.168.2.118:48484/v2/local/device-info
[Debug]   Found build tasks:
[Debug]       wifi-connect: image pull [balena/wifi-connect:aarch64]
[Debug]   Resolving services with [jetson-nano|aarch64]
[Debug]   Found project types:
[Debug]       wifi-connect: External image
[Debug]   Probing remote daemon for cache images
[Debug]   Using 7 on-device images for cache...
[Debug]   Starting builds...
[Debug]   Setting device state...
[Debug]   Sending request to http://192.168.2.118:48484/v2/local/target-state
[Debug]   Sending target state: {"local":{"name":"local","config":{"SUPERVISOR_POLL_INTERVAL":"60000","SUPERVISOR_INSTANT_UPDATE_TRIGGER":"true","SUPERVISOR_LOCAL_MODE":"true","SUPERVISOR_CONNECTIVITY_CHECK":"true","SUPERVISOR_LOG_CONTROL":"true","SUPERVISOR_DELTA":"false","SUPERVISOR_DELTA_REQUEST_TIMEOUT":"30000","SUPERVISOR_DELTA_APPLY_TIMEOUT":"0","SUPERVISOR_DELTA_RETRY_COUNT":"30","SUPERVISOR_DELTA_RETRY_INTERVAL":"10000","SUPERVISOR_DELTA_VERSION":"2","SUPERVISOR_OVERRIDE_LOCK":"false","SUPERVISOR_PERSISTENT_LOGGING":"false","HOST_FIREWALL_MODE":"","HOST_DISCOVERABILITY":"true","SUPERVISOR_VPN_CONTROL":"false"},"apps":{"1":{"name":"localapp","commit":"localrelease","releaseId":"1","services":{"1":{"environment":{},"labels":{"io.balena.features.dbus":"1","io.balena.features.firmware":"1"},"restart":"always","network_mode":"host","privileged":true,"imageId":1,"serviceName":"wifi-connect","serviceId":1,"image":"local_image_wifi-connect:latest","running":true}},"volumes":{},"networks":{}}}},"dependent":{"apps":[],"devices":[]}}
[Debug]   Sending request to http://192.168.2.118:48484/v2/local/target-state
[Info]    Streaming device logs...
[Debug]   Sending request to http://192.168.2.118:48484/v2/local/device-info
[Live]    Watching for file changes...
[Live]    Waiting for device state to settle...
[Debug]   Device state not settled, retrying in 1000ms
[Logs]    [4/13/2021, 5:45:25 PM] Installing service 'wifi-connect sha256:1d7bdc214cd9351a326e440229aee10f73dcb0778b5be4c9e17f6388d94d6002'
[Logs]    [4/13/2021, 5:45:25 PM] Installed service 'wifi-connect sha256:1d7bdc214cd9351a326e440229aee10f73dcb0778b5be4c9e17f6388d94d6002'
[Logs]    [4/13/2021, 5:45:25 PM] Starting service 'wifi-connect sha256:1d7bdc214cd9351a326e440229aee10f73dcb0778b5be4c9e17f6388d94d6002'
[Logs]    [4/13/2021, 5:45:26 PM] Started service 'wifi-connect sha256:1d7bdc214cd9351a326e440229aee10f73dcb0778b5be4c9e17f6388d94d6002'
[Debug]   Device state not settled, retrying in 1000ms
[Live]    Device state settled
[Logs]    [4/13/2021, 5:45:31 PM] [wifi-connect] Checking internet connectivity ...
[Logs]    [4/13/2021, 5:45:32 PM] [wifi-connect] Your device is already connected to the internet.
[Logs]    [4/13/2021, 5:45:32 PM] [wifi-connect] Skipping setting up Wifi-Connect Access Point. Will check again in 120 seconds
[Logs]    [4/13/2021, 5:47:32 PM] [wifi-connect] Checking internet connectivity ...
[Logs]    [4/13/2021, 5:47:33 PM] [wifi-connect] Your device is already connected to the internet.
[Logs]    [4/13/2021, 5:47:33 PM] [wifi-connect] Skipping setting up Wifi-Connect Access Point. Will check again in 120 seconds
[Warn]    Connection to device lost
Retrying "Streaming logs" after 1.0s (1 of 1000) due to: DeviceConnectionLostError: Connection to device lost
Retrying "Streaming logs" after 2.0s (2 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (3 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (4 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (5 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (6 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (7 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (8 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (9 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (10 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (11 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (12 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (13 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (14 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (15 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (16 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (17 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (18 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (19 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (20 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (21 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (22 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (23 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (24 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (25 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (26 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (27 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
Retrying "Streaming logs" after 2.0s (28 of 1000) due to: Error: connect ETIMEDOUT 192.168.2.118:48484
[Live]    Cleaning up device...
[Debug]   Cleaning up done.
Task aborted on SIGINT signal
SIGINTError: Task aborted on SIGINT signal
    at process.sigintHandler (/usr/local/lib/balena-cli/build/utils/helpers.js:315:20)
    at Object.onceWrapper (events.js:421:26)
    at process.emit (events.js:314:20)
    at process.EventEmitter.emit (domain.js:483:12)

$ balena push balena.local --debug
[debug] new argv=[/usr/local/lib/balena-cli/bin/node,/usr/local/lib/balena-cli/bin/run,push,balena.local] length=4
[Debug]   Using build source directory: .
[Debug]   Pushing to local device: balena.local
[Debug]   Checking we can access device
[Debug]   Sending request to http://192.168.2.120:48484/ping
ECONNRESET: socket hang up
Error: socket hang up
    at connResetException (internal/errors.js:609:14)
    at Socket.socketOnEnd (_http_client.js:458:23)
    at Socket.emit (events.js:326:22)
    at Socket.EventEmitter.emit (domain.js:483:12)
    at endReadableNT (_stream_readable.js:1241:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21)
For further help or support, visit:
https://www.balena.io/docs/reference/balena-cli/#support-faq-and-troubleshooting

Thanks for the great logs and troubleshooting here @codethief … and, you’re right about those ealier thoughts on JetPack 4.5 being the issue. In the case of the Nano, yep, it literally won’t even boot. So looking at this, just to clarify, did you actually upgrade the Supervisor, or not yet? It can be done via the drop-down menu when you are on a device’s details page in balenaCloud.

So looking at this, just to clarify, did you actually upgrade the Supervisor, or not yet? It can be done via the drop-down menu when you are on a device’s details page in balenaCloud.

No, I did not. Thanks for pointing this out!

So I just did the upgrade (following this post, I ran $ update-resin-supervisor -t v12.5.10 in a host shell on my device since I wanted to stay in local mode). So far, things are looking very good! The first balena push is working without problems and curl localhost:48484 hasn’t shown any hanging or timeout, either!

My colleague just did the same and it seems to be working for him, too! Looks like it was indeed the supervisor then!

Once again, thank you for your help!

Question: Are images for the Jetson devices released manually by your team or is there an automatic build process involved? In any case: Could you update the Xavier image with the updated supervisor? :slight_smile: (In fact, now that I’m thinking about it: Would a supervisor upgrade via balenaCloud even work if the supervisor’s API is unreachable?)

Hi, the device OS releases are done manually after validation. The v12.5.10 supervisor has just been merged into meta-balena v2.73.9 and I have created a request for a Xavier build (Request for a Xavier 2.73.9 release · Issue #159 · balena-os/balena-jetson · GitHub) to prioritize its release and linked it to this ticket so you will be notified once it’s closed.

About your last question, dashboard supervisor updates do not rely on the supervisor API so it will work even if the API is unreachable.

2 Likes