Nvidia Jetson Xavier DevKit eMMC bringup issues

Product support balenaOS
1

I’m a first time Balena user, evaluating whether it can be used to manage Jetson Xavier devices.

I created a balenaCloud account, downloaded Jetson Xavier NX Devkit eMMC version 2.88.4+rev10 (development), rebooted the Xavier into recovery mode, and ran the jetson-flash tool:

git clone https://github.com/balena-os/jetson-flash.git
git checkout a53f3e7a4d52b9c3752dc19ad9d7dc9d419ecbc7
cd jetson-flash
npm install
./bin/cmd.js -f balena-cloud-experimental-jetson-xavier-nx-devkit-emmc-2.88.4+rev10-v12.11.0.img -m jetson-xavier-nx-devkit-emmc

This appeared to complete successfully, ending with “The target t186ref has been flashed successfully.” I’ve attached the full log below.

I rebooted the Jetson and did not see it appear in my balenaCloud dashboard.
sudo balena scan also shows no results on the local network.

When I plug in an HDMI cable, the factory-installed NVIDIA Jetpack desktop application still appears to be on the device.

Any help is much appreciated.

2022-04-13.log (63.4 KB)

2

To clarify, this is using the Jetson Xavier sold here:

which upon further reading I understand to be a production Jetson Xavier NX module.

I wondered if my issue had to do with using the devkit image.

So I tried again using the Jetson Xavier (development) image:
./bin/cmd.js -f balena-cloud-experimental-jetson-xavier-2.88.4+rev12-v12.11.0.img -m jetson-xavier

but that failed: Failed flashing t186ref.

Which is the intended image to use with the Jetson Xavier NX production module?

3

Hi, thanks for your message. The default images for Jetson Xavier images correspond to the Nvidia development kits:

  • Nvidia Jetson Xavier, this is the Jetson Xavier AGX development kit
  • Nvidia Jetson Xavier NX Devkit, these correspond to the SDcard and eMMC versions

To boot these images you first need to make sure that your development kit is programmed with a L4T version image that is compatible with the balena image, as the existing bootloader needs to boot the balena image.

As you are speaking about supporting a different hardware than a development kit (https://www.seeedstudio.com/Jetson-SUB-Mini-PC-Blue-p-5212.html) it is possible that the development kit images will not work out of the box. Typically though, a different kernel device tree is all that is needed to support a different carrier board.

Usually the development kit image boots and registers with balenaCloud just fine, and the dashboard can be used to specify a different device tree that is customized for the new carrier board. This new device tree will need to be contributed by you as a pull request to the balena-jetson repository so that it can be included as part of the default images.

In summary, you will need to debug the development kit image to understand why it does not boot and connect to the cloud. I would suggest you start by making sure the board is programmed with an Nvidia Jetson L4T image that is compatible with the balena version you want to use. For the latest v2.88.4+rev11 release available, this is L4T 32.6.1 available with Jetpack 4.6.

I also suggest you build a custom development image as described in GitHub - balena-os/balena-jetson, and use the -d flag to barys so the project is configured with the OS_DEVELOPMENT variable in conf/local.conf. This will enable serial output from the bootloader output which may be useful.

Once you are booting a development image and are able to log into the shell, please provide the output of journalctl --no-pager so we can try to help debugging.

7

Hi @ijsijsijsijsijsijs adding to what my colleague already explained, I have a question about what you mentioned in your first post:

When I plug in an HDMI cable, the factory-installed NVIDIA Jetpack desktop application still appears to be on the device.

Do you mean that after the device is flashed with jetson-flash, the board still boots Ubuntu? If so, can you please post the UART boot logs?

The product you linked appears to include a " Pre-installed 128GB (M.2 key M) SSD" and I suspect that Nvidia’s Cboot bootloader could be loading the OS from that drive instead of booting balenaOS from the eMMC.

9

Thank you for the quick responses, really appreciate it!

Do you mean that after the device is flashed with jetson-flash, the board still boots Ubuntu? If so, can you please post the UART boot logs?

Yes. Of course, I will post later today.

The product you linked appears to include a " Pre-installed 128GB (M.2 key M) SSD " and I suspect that Nvidia’s Cboot bootloader could be loading the OS from that drive instead of booting balenaOS from the eMMC.

That seems like a good hypothesis, good eye. I’ll try to do some more reading on this.

10

I had to order a USB-to-TTL serial cable to get UART boot logs, but that’s en route.

@alexgg I did confirm that L4T is 32.6.1.

nv_tegra_release.log (114 Bytes)

I think you’re right @acostach that the OS is being loaded from the SSD. I see Resin partitions on /dev/mmcblk0p1 Here is the output of lsblk -f, and a couple of screenshots:

nvme0n1p1
nvme0n1p11483×913 90.9 KB

lsblk.log (2.7 KB)

11

mmcblk0p1
mmcblk0p11485×910 86.8 KB

12

I am able to connect to the serial console with a micro-USB cable from my Ubuntu workstation, with minicom -D /dev/ttyACM0 -8 -b 115200 -C minicom.log. However I don’t see any boot logs on power cycle of the Jetson.

I’m not sure whether this is because:

  • I haven’t set the OS_DEVELOPMENT variable, as @alexgg described
  • The micro-USB interface to the serial console is only enabled after boot
  • Some other issue

I’m guessing it’s the first. I’ll work on building that custom development image. Thanks for bearing with me, new to this.

13

@ijsijsijsijsijsijs the boot logs are visible with the USB-to-TTL serial cable connected to the TX, RX, GND pins, at least on the devkit. Not sure if these pins are exposed on your device though. If you have access to the SSD you can simply disconnect it and the device should boot from the eMMC.

15

@acostach @alexgg thank you for your patience while I gathered the boot logs. They’re attached below:

minicom.log (38.7 KB)

I saw message in the logs about boot order, and it appeared to be booting from the nvme SSD, as you suspected. I followed the instructions here to temporarily change the boot order:

setvar boot-order emmc,sd,usb,nvme,net
boot

which booted me into BalenaOS. I now see the device in my cloudBalena dashboard. Thank you!

18

Thank you for informing us about this type of problem @ijsijsijsijsijsijs . BalenaOS can only be loaded from the same medium on which the tegra bootloaders reside, be it sd-card for Xavier NX SD-CARD or the eMMC for the other type of module. I’ve raised this github issue: Cboot should use device-specific boot medium as default · Issue #310 · balena-os/balena-jetson · GitHub and we will modify cboot to load the medium that was written with jetson-flash by default. We’ll get back to you when this becomes available in Balena cloud.

20

Hi, just wanted to let you know that this issue has been addressed in the latest v2.98.33 image releases for the Xavier, and that the latest jetson-flash 0.5.26 should be used to ensure that the sd-card or the eMMC are used for booting the image.

1 Like