In our attempts to build a custom BalenaOS for the Auvidea JN30B carrier board as in following PR we have managed to successfully build an image with the help of @acostach. However, when flashing the board using jetson-flash I’m not sure if the board specific .dtb file is flashed. It seems to populate the rootfs with the default tegra210-p3448-0002-p3449-0000-b00.dtb as shown in the logs enclosed. jetson-flash.log (131.8 KB)
Flash completes successfully but BalenaOS is not booting. I’ve ordered some connectors in order to debug with TTL console and awaiting delivery. In the meantime, anybody able to help?
Thanks for the fast response and your suggestions, really appreciate your help. I’ve tried flashing the device with the proposed branch and saved the output jetson-flash.log (129.9 KB). However, tool still seems to use default .dtb.
From the latest log I see the partitions RP1 and DTB are being used, which is good:
[ 2.0780 ] Writing partition RP1 with RP1.encrypt
[ 2.1133 ] […] 100%
[ 2.3391 ] Writing partition DTB with DTB.encrypt
[ 2.3751 ] […] 100%
Now, can you please check in your yocto build if the generated files RP1 and DTB are actually the dtb that you are using for this particular carrier board?
You should be able to see RP1 and DTB in the build directory, in /tmp/work/jetson_nano-poky-linux/resin-image/1.0-r0/resin-boot/bootfiles/
At this point, they are signed dtbs, so you can’t really use dtc to decompile it, but if you know some strings that only occur in your particular dtb, you can quickly identify if it’s the right dtb or not ( i.e strings RP1 / DTB | grep particular_string_in_my_dtb).
Good suggestion. I’ve converted both tegra210-p3448-0002-p3449-0000-b00.dtb files from NVIDIA and AUVIDEA a while ago and saved them to .dts. Here I noticed the ‘auvidea’ string in the file path of nvidia,dtsfilename
Thanks for confirming, now we need to understand what happens more exactly, to confirm that on the board emmc the default dtb is present. Are you able to login on the board somehow, from dashboard or maybe via SSH? Or if you are using a development image and the board connects to local network via ethernet and running a “balena scan” to see the device?
We need a way to check if the partitions on the board have this particular string (i.e strings /dev/disk/by-label or by-partlabel/RP1. If you want you can send me a temporary link to the image you are flashing, I will flash it on the devkit and inspect the resulting sd-card image.
One more thing that probably is not be related but worth mentioning: the current build is based on 32.2 and the dtb appears to be for 32.2.1.
Currently we have 6x JN30B Rev 4. boards with Nano production modules at the office. After flashing, BalenaOS doesn’t boot at all and doesn’t get an IP-address assigned. In order to debug I’ve ordered the necessary JST GH connector such I can attach a serial console using a TTL converter. These parts will come in tomorrow.
I’ve noticed as well. However, AUVIDEA released the firmware for this L4T release. Besides the compiled .dtb files they also supply the changes in source .dts files as in jn30b firmware. Probably more work, we could also patch the default kernel instead by using the sources.
I’ve flashed the image you provided on both the emmc and the sd-card versions, mounted on the default devkit baseboard.
I can confirm that partitions RP1 and DTB get flashed with the correspondent partitions from the image, in this case with the auvidea dtb. Here is an example where I took out the sd-card after flashing with jetson-flash and inspected RP1 and DTB:
Thanks for checking, good to know that flashing seems to function properly. However, do I understand correctly once you flash for Nano emmc the OS doesn’t take over after u-boot?
With the image with auvidea dtb the most logs I get are until u-boot is loaded. I don’t see what happens afterwards. With a nano image from dashboard, after flashing emmc, OS is booted and I can login. But this is quite expected, because I’m using the devkit baseboard.
It seems that similar things are happening with the jn30b, what could be the reason? I will check the logs after flashing the jn30b once I get the necessary connectors and get back to you!
Hmm, not sure what the cause would be… If you flash the board using only the default 32.2 BSP - not 32.2.1, no Balena image, no jetson-flash, only 32.2 BSP - and replace the dtb with the one provided for jn30b are you able to get past the current point in the logs? Do you see u-boot trying to load extlinux file or the kernel,?
As I understand, that process is a bit different, in the sense it’s probably not using the dtb from the image, but one replaced in the cached BSP?
So, on your side, 32.2 BSP just with the dtb replaced, flashing with just flash.sh from BSP archive, does it get you to see the default kernel booting?
I received the necessary connectors in order to attach a TTL serial console. When flashing the JN30B with JetPack 4.2.2 together with the supplied firmware of Auvidea I only get console logs after Ubuntu has booted and the login prompt is popping up.
When flashing the JN30B board with custom BalenaOS as we developed last weeks I don’t get console logs at all simply because the boot process doesn’t reach to boot BalenaOS.
However, when flashing the Nano devkit with JetPack 4.2.2 i receive console logs from u-boot all the way until login prompt of Ubuntu.
I’m a bit clueless on why this is happening but it seems it has something to do with the hardware. I will try to contact Auvidea to support on the matter. In the meantime, do you have any suggestions?
Good, so 4.2.2 + Auvidea firmware works fine. That’s expected. But what about 4.2, L4T 32.2.0, the one I was mentioning earlier?
To narrow down the root cause, I would check in the following order:
Check if L4T 32.2.0 + Auvidea firmware works. Just BSP + firmware, no BalenaOS. Not to be mistaken with 32.2.1 that you already checked.
If step 1) works, flash again, but this time only replace the device tree with the one you are using for Auvidea and the same one that you added to BalenaOS. If it doesn’t work, then start replacing with some more Auvidea files and test again until you start seeing console logs.
Also, please don’t unpack the whole Ubuntu archive to rootfs directory, because it will take a lot longer to flash. Seeing u-boot and some kernel logs should be enough to determine that you found what was missing.
For BalenaOS currently you have standard nvidia BSP + auvidea device tree. Therefore by testing with standard BSP and replacing with Auvidea files, adding one by one each time, you can see where booting starts to work.
Thank you for testing and providing this very in-depth analysis @remsol . Indeed, your results from points 3) and 5) were very helpful for this investigation.
It looks like there was a change in u-boot-tegra upstream that switched the environment location, thus breaking boot.
I have updated the jn30b branch to exclude the breaking upstream change, and now I’m able to also boot the image for jn30b on the standard devkit:
Thanks for your expert support. Hereby I can confirm a booting BalenaOS image on the Auvidea JN30B board. Probably, it would have cost me a long time to find this problem myself. Although I get the upstream change is messing up things could you please explain why this is only happening in our custom build? We should see similar behaviour in (5) right?
Besides, I noticed some things when building the image which I hope you can explain me. First of all the following folder shows several .dtb files. Is it correct to say that the kernel is first build from NVIDIA sources and subsequently the .dtb is replaced in the appropriate partitions?
ls build/tmp/deploy/images/jn30b-nano/tegra210*
tegra210-p3448-0000-p3449-0000-a02--4.9+git0+3dcbed5d27-r0-jn30b-nano-20191111161247.dtb
tegra210-p3448-0000-p3449-0000-a02-jn30b-nano.dtb
tegra210-p3448-0000-p3449-0000-a02.dtb
tegra210-p3448-0002-p3449-0000-b00-jn30b-JP4.2.2.dtb
During build I’ve noticed some warnings;
WARNING: popt-native-1.16-r3 do_fetch: Failed to fetch URL http://rpm5.org/files/popt/popt-1.16.tar.gz, attempting MIRRORS if available
WARNING: networkmanager-1.20.2-r0 do_patch: Fuzz detected:
Applying patch 0002-Do-not-create-settings-settings-property-documentati.patch
patching file Makefile.am
Hunk #1 succeeded at 1429 with fuzz 1 (offset 131 lines).
Hunk #2 succeeded at 4082 (offset 154 lines).
patching file configure.ac
Hunk #1 succeeded at 1180 (offset -35 lines).
The context lines in the patches can be updated with devtool:
devtool modify networkmanager
devtool finish --force-patch-refresh networkmanager <layer_path>
Don't forget to review changes done by devtool!
WARNING: networkmanager-1.20.2-r0 do_patch: QA Issue: Patch log indicates that patches do not apply cleanly. [patch-fuzz]
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_IP_NF_TARGET_LOG=m in the kernel configs failed for misc.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_CRYPTO_LZO=m in the kernel configs failed for zram.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NFT_FIB_INET=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NFT_FIB_IPV6=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NFT_OBJREF=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NF_TABLES_IPV4=y in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NF_TABLES_ARP=y in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NF_TPROXY_IPV4=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NFT_SOCKET=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NFT_FIB=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NFT_OSF=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NFT_CONNLIMIT=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NFT_TUNNEL=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NF_TABLES_SET=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NF_TPROXY_IPV6=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NFT_FIB_NETDEV=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NFT_FIB_IPV4=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NF_TABLES_INET=y in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NF_SOCKET_IPV4=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NF_SOCKET_IPV6=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NFT_TPROXY=m in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NF_TABLES_IPV6=y in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_NF_TABLES_NETDEV=y in the kernel configs failed for nf_tables.
WARNING: linux-tegra-4.9+gitAUTOINC+3dcbed5d27-r0 do_kernel_resin_checkconfig: Checking for CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF=y in the kernel configs failed for debug_kmemleak.
NOTE: Tasks Summary: Attempted 3801 tasks of which 10 didn't need to be rerun and all succeeded.
Except these warnings the build succeeded successfully, you can find the resulting image here. I flashed the board with jetson-flash (branch jetson_nano_use_rp1) after which BalenaOS boots and I’ve logged the serial output auvidea-jn30b-balenaos.log (74.4 KB). By quickly examining the logs I don’t see major issues. After the first boot I’ve booted twice again, this time with CSI camera connected at each channel which are showing up in /dev/video0 in HostOS.
In order to test the application side of things we have 4 products ready to be flashed and installed at one of our pilot locations. Could you explain me the procedure in order to onboard in BalenaCloud? Correct me if I’m wrong but the PR needs to be merged into master before BalenaCloud is accepting this new board? Once this is done, what do you expect from us in order to test and maintain the image for future BalenaOS updates?
