Installation of Balena0S on Nvidia Orin NX stuck at the end

Hello,

I have installed BalenaOS successsfully in the past on a RaspberryPI and on a Jetson Xavier AGX.

Nonetheless, I am stuck with the flashing process on a Nvidia Orin NX. It’s not exactly the devkit but it is embedded in a syslogic computer. However, the process should be quite similar because they refer the the nvidia documentation of the devkit to explain how to flash the computer again.

I have tried to follow the instructions frm the github repository. I am able to place the computer in recovery mode, to start and run the docker, the flashing process does not seem to show critical issue. Finally, the LED switches off after around 15 minutes. But when I switch my board off and on, I am still on the previous OS version, BalenaOS is not present. I tested on the Nvidia default dev kit and on another PC based on the orin NX module but I face the same issue on both

I really struggle to understand what is going on, since I cannot find a way to show the result of the flashing process from the USB to the Jetson QSPI memory.

Have anyone encountered the same problem and do you have any idea on how to get out of this? I thought that versions could be an issue but I am not sure.

  • Host PC: Ubuntu 22.04.4 LTS (6.5.0-35-generic)
  • Docker version 26.1.3, build b72abbb
  • Hardware: Nvidia Jetson Orin NX 16GB in a SYSLOGIC RML A4NX compter
  • BalenaOS: 5.3.10 (I cannot find on which L4T this is based)
  • Jetson-flash: on master branch

Thanks you!

1 Like

Hello @gurbain first of all welcome to the balena community!

What device type did you test on balena?

I’m using NVIDIA Jetson Orin NX 16GB from Seeed Studio which is this device type Seeed reComputer J4012 16GB.

Hi @mpous,

I have tried with this device: ZED Box - Orin NX 16GB for Edge AI | Stereolabs
And this one: Rugged Computer RPC RML A4NX (GMSL)

With similar (non-conclusive) results. I know they are not the official dev kits but they are based on the same NX module and at least the later allow firmware update with the official nvidia NX devkit device tree and image.

I don’t know where to start since I’m unable to see the logs about what’s going on as soon as the memory transfer from the USB key to the PC happens.

@gurbain i understand your pain! I also had similar issues with the reComputer.

Are you using those devices for any specific requirement?

Do these manufacturers have the bsp layers for Yocto or any specific dtoverlay or dtparams?

I pinged internally the devices team to see how we can help you more!

Hi @gurbain , regarding this:

Could there be more than 1 internal storage present in the Syslogic board?

Since the device appears to boot the USB flasher image and turns off after provisioning is completed, it could be that there is a NVME present and it is flashed, but it then continues to boot from another NVME, SSD, SD-CARD etc, which has Ubuntu flashed on it. You could check from the previous OS, if this is the case.

I also recommend checking if the devices have any debug UART port to collect logs during flashing. That might require to open the device’s case.

Once you get access to the debug uart, you can use a Development image for provisioning, and login with user root to check if flashing works using: journalctl | grep flasher
and
systemctl status resin-init-flasher.

Hi,

@mpous : yes, I use it mainly because of their GMSL2 and CAN connection and the second one also for its mechanical properties (IP protection, casing, thermal, EM, …).
No, I could not find any BSP layers for it. I know someone required it on for the stereolabs device on the stereolabs forum but no real answer there.

@acostach: that’s actually a very interesting comment. I did not think about double memory… About the UART, are we talking about UART0 Interface of the Jetson ORIN NX Module (I can have access to this one) or another one?

Unfortunately, I will be away from these computer for the next 2-3 weeks but I will test this, afterwards, as soon as I can and post it if I find anything.

1 Like

Keep us posted @gurbain !

Hi,

I’m also trying to flash BalenaOS on a Stereolabs Orin NX 16GB, without success so far.

The flashing process from the Docker container doesn’t show any error, and ends with the RCM-boot and the message telling to wait 10-15min for the internal flashing process to complete, however in my case the power LED never turns on. I still waited about 15min but the Orin NX wasn’t flashed, it’s still booting on the previous OS.

Stereolabs has custom scripts to flash this device, downloadable from their docs.

Looking at the script for NVIDIA Jetpack 5.1.2 / L4T 35.4.1 (stereolabs.sfo2.cdn.digitaloceanspaces .com/utils/zed_boxes/zedbox_onx_usb_flash_5.1.2.sh), on top of the stock rootfs it applies the following patches:

  • HDMI patch: stereolabs.sfo2.cdn.digitaloceanspaces .com/utils/zed_boxes/onx/35.4/orin_nx_hdmi_gpio_fix_l4t_354.patch
  • Replaces the kernel image with this one: stereolabs.sfo2.cdn.digitaloceanspaces .com/utils/zed_boxes/onx/35.4/Image
  • Replaces tegra234-p3767-0000-p3509-a02.dtb with this one: stereolabs.sfo2.cdn.digitaloceanspaces .com/utils/zed_boxes/onx/35.4/tegra234-p3767-0000-p3509-a02.dtb

The script for NVIDIA Jetpack 6.0 / L4T 36.3 (stereolabs.sfo2.cdn.digitaloceanspaces .com/utils/zed_boxes/zedbox_onx_usb_flash_6.0.sh) is a little bit different and does not replace the kernel image, instead it:

  • Adds/replaces these device tree files: stereolabs.sfo2.cdn.digitaloceanspaces .com/utils/zed_boxes/zedbox_device_trees_363.tar
  • Adds some kernel modules to the rootfs: stereolabs.sfo2.cdn.digitaloceanspaces .com/utils/zed_boxes/zedbox_rootfs_363.tar (this seems to be for the WiFi, which I don’t care about)

Unfortunately, I’m not very familiar with embedded systems development. But presumably I would need to apply the custom DTB in some way?

As an aside I also tried flashing a Yocto build based on meta-tegra using the initrd-flash script (following these docs: initrd flashing support · OE4T/meta-tegra Wiki · GitHub) however the flashing process gets stuck on “Waiting for USB storage device”, similar to what is described in this discussion: Issues with flashing jetson orin nx 16GB · OE4T/meta-tegra · Discussion #1304 · GitHub (but I believe what is discussed there doesn’t necessarily apply to the specific case of a Stereolabs Orin NX).

Thanks

PS: when posting it told me I could only include 5 links because I’m a new user, so I had to intentionally break some links, I added a space before “.com”…

2 Likes

Thanks for sharing @tdoubliez and welcome to the balena community!

I will connect with the balena devices team to explore how we can help you more!

Feel free to share more!

1 Like

HI @tdoubliez, is there any way to connect to a debug UART on this device to obtain boot logs? I didn’t spot anything on the link you sent and the micro-USB port appears to be used for flashing, but perhaps there are debug UART pins exposed if the case is taken off. I recommend checking with the manufacturer first, to avoid voiding the warranty.

Thanks @mpous

@acostach Opening the box will void the warranty so I haven’t done that for now. I used the micro-USB port for flashing, and the device automatically boots into recovery mode when the micro-USB cable is connected.

My goal is to make it easy to provision a fleet of those boxes with the software I need to run on it, and have a way to perform OTA updates. I’m investigating multiple parallel paths to achieve that:

  • Yocto build based on meta-tegra and meta-aws to add AWS IoT/Greengrass components and then deploy software using Greengrass + use an ACM client cert for authentication with some backend APIs
  • BalenaOS with balenaCloud to provision software, but also using AWS IoT/ACM for client cert authentication to APIs
  • Because of the uncertainty of getting one of the above two options to work on the custom Stereolabs box, the third option that I’m looking at is to use the Stereolabs script, which is based on the official NVIDIA Jetson flashing script, that I’ll customize to install Greengrass Core and have the device auto-provision itself on first boot in AWS IoT

The convenience of BalenaOS is appealing, but even if I would manage to flash it on the box, I think there will be other hurdles after that.

The thing is that my application involves the ZED X camera and I will also need the following installed on the Orin NX:

  • NVIDIA JetPack
  • ZED SDK
  • GMSL2 driver
  • NVIDIA container toolkit (the application will be containerized and needs access to the GPU)