UART Failure after upgrading to 2.71.3+rev1 from 2.50.1+rev1 on Variscite IMX8M

Product support balenaOS
,
61

Hi @anathan84, thanks for confirming that the kernel 5.4.85 solves this issue for you. I’ve opened a github ticket to track your request, you can keep an eye on it for updates.

63

Any guess on timing of the update? Else I may try bitbaking the new version on top of your yocto layers locally.

64

@anathan84 the item is in our queue but we can’t offer a release estimate for it right now.

66

Hi @anathan84 , have you bitbaked the latest version locally, did it work as expected? If you haven’t checked yet, we have a WIP PR that updates to Dunfell + kernel 5.4.85 and which you can build locally with:

git checkout set_state_to_released
git submodule update --init -recursive
balena-yocto-scripts/build/barys -b build_mm -m imx8mm-var-dart

Let us know how it works.

67

Hi @anathan84 @adamshapiro0 , the v2.80.5+rev2 BalenaOS image in the dashboard for the DART MX8M Mini is based on Dunfell and has the kernel 5.4.85. Let us know how it works for you.

69

Great thank you for the follow up. Well get this tested in our next sprint.

70

Just to update the thread, we’ve been unable to get he 2.80.5 official build to run on our hardware. We tried a fresh install with SD Card which appears to be getting jammed up with u-boot (using the exact same hardware with 2.50 and 2.7 work fine). We also tried upgrading a device using the web console from 2.50 to 2.80 and the system failed to boot. I’ve attached the log of that here.
upgrade failure.txt (17.1 KB)

71

Interesting, the same image boots fine on the hardware we have. In your logs I see you have the VSM-DT8MM-202A, whereas we have the:

Part number: VSM-DT8MM-102
...
Production date: 2020 Jan 15
...
Booting from mmc ...
fdt_file=imx8mm-var-dart-dt8mcustomboard-legacy.dtb
45402 bytes read in 26 ms (1.7 MiB/s)

Can you stop u-boot in cmdline by pressing space or esc in the serial console of the development image, and load the non-legacy dtb to check if the device boots with it? With the -102 and non-legacy device tree the pcie driver prints a few errors but the OS boots successfully, would be interesting to see if it works the other way around.

For example:

u-boot=> editenv findfdt
edit: if test $fdt_file = undefined; then if test $board_name = VAR-SOM-MX8M-MINI; then if test $carrier_rev = legacy; then setenv fdt_file imx8mm-var-som-symphony-legacy.dtb; else setenv fdt_file imx8mm-var-som-symphony.dtb; fi; else if test $carrier_rev = legacy; then setenv fdt_file imx8mm-var-dart-dt8mcustomboard.dtb; else setenv fdt_file imx8mm-var-dart-dt8mcustomboard.dtb; fi; fi; fi;
u-boot=> run bootcmd

72

I must be doing something odd, I changed the env variable but the script still prints out the legacy dtb. See below:

u-boot=> editenv findfdt
edit: if test $fdt_file = undefined; then if test $board_name = VAR-SOM-MX8M-MINI; then if test $carrier_rev = legacy; then setenv fdt_file imx8mm-var-som-symphony-legacy.dtb; else setenv fdt_file imx8mm-var-som-symphony.dtb; fi; else if test $carrier_rev = legacy; then setenv fdt_file imx8mm-var-dart-dt8mcustomboard.dtb; else setenv fdt_file imx8mm-var-dart-dt8mcustomboard.dtb; fi; fi; fi;
u-boot=> run bootcmd
Scanning mmc devices 0 1 2
Card did not respond to voltage select!
Found resin image on mmc 2
Loading resinOS_uEnv.txt from mmc device 2 partition 1
38 bytes read in 5 ms (6.8 KiB/s)
Import resinOS_uEnv.txt in environment
Loading extra_uEnv.txt from mmc device 2 partition 1
Loading bootcount.env from mmc device 2 partition 1
13 bytes read in 5 ms (2 KiB/s)
Import bootcount.env in environment
WARNING! BOOTLIMIT EXCEEDED. SWITCHING TO PREVIOUS ROOT
WARNING! was: resin_root_part=3
WARNING! now: resin_root_part=2
gpio: pin 137 (gpio 137) value is 1
switch to partitions #0, OK
mmc2(part 0) is current device
11069394 bytes read in 58 ms (182 MiB/s)
Uncompressed size: 24745992 = 0x1799808
Booting from mmc ...
fdt_file=imx8mm-var-dart-dt8mcustomboard-legacy.dtb
WARN: Cannot load the DT
u-boot=>

Admittedly this is not something I’ve done before, so maybe I’m screwing up the basic syntax of editenv?

73

Also tried this:

u-boot=> setenv fdt_file imx8mm-var-som-symphony.dtb
u-boot=> run bootcmd
Scanning mmc devices 0 1 2
Card did not respond to voltage select!
Found resin image on mmc 2
Loading resinOS_uEnv.txt from mmc device 2 partition 1
38 bytes read in 5 ms (6.8 KiB/s)
Import resinOS_uEnv.txt in environment
Loading extra_uEnv.txt from mmc device 2 partition 1
Loading bootcount.env from mmc device 2 partition 1
13 bytes read in 4 ms (2.9 KiB/s)
Import bootcount.env in environment
WARNING! BOOTLIMIT EXCEEDED. SWITCHING TO PREVIOUS ROOT
WARNING! was: resin_root_part=3
WARNING! now: resin_root_part=2
gpio: pin 137 (gpio 137) value is 1
switch to partitions #0, OK
mmc2(part 0) is current device
11069394 bytes read in 58 ms (182 MiB/s)
Uncompressed size: 24745992 = 0x1799808
Booting from mmc ...
fdt_file=imx8mm-var-som-symphony.dtb
WARN: Cannot load the DT
u-boot=>

No luck either.

74

Looks like you are rebooting the device right after updating the HostOS, u-boot notices the device rebooted before the healthchecks in the new OS completed, and it proceeds to switch to loading the dtb and kernel from the old OS. And since in the old OS the dtb has a different name, you get the error:

WARN: Cannot load the DT

This mechanism is used to keep the device accessible in case of potential boot issues in the new OS. Therefore, you need to either wait for at least 3-4 minutes before rebooting the board after the HostOS update, or, flash the board directly with the new OS.

About the dtb, you can also try set it directly:

setenv fdt_file imx8mm-var-dart-dt8mcustomboard.dtb

it shouldn’t get overridden:

if test $fdt_file = undefined; then if ...
76

Hey Aaron

Were you able to try the things recommended by acostach in the previous message?

78

Maybe?

I tried the device again today, it appears to get further but stops before booting on this line:

[ 2.085491] [drm] Initialized imx-drm 1.0.0 20120507 for soc@0:bus@32c00000:display-subsystem on minor 0

Full log from boot is attached here.
bootload.txt (16.9 KB)

I don’t know why it’s saying this early on:
MMC card init failed!
** Block device MMC 1 not supported

I’m assuming MMC0 is the embedded flash on the variscite module, MMC2 is the SDHC card, I have no clue what MMC1 is.

79

Hi Aaron, in the last log you attached you’re still loading the -legacy dtb in the latest logs. Did you try boot with the non-legacy dtb, did it work?

I suspect you’re using a custom carrier board that differs from the Varsicite Devkit DT8M-Customboard v1.X, at least with regards to pcie, but you need to confirm this. So can you please try disable pcie from the device tree and see if the device boots?

For this you need to write the sd-card with the latest image, then plug the sd-card in your PC, decompile the legacy dtb:

dtc -I dtb -O dts /media/acotsach/flash-rootA/boot/imx8mm-var-dart-dt8mcustomboard-legacy.dtb -o temp.txt

Edit temp.txt and change status from “okay” to status="disabled" on the pcie@33800000 node:

pcie@33800000 {
			compatible = "fsl,imx8mm-pcie", "snps,dw-pcie";
			reg = <0x33800000 0x400000 0x1ff00000 0x80000>;
			reg-names = "dbi", "config";
			...
			status = "disabled"

Save and then compile the dts from source:
dtc -I dts -O dtb temp.txt -o imx8mm-var-dart-dt8mcustomboard-legacy.dtb

Replace the old dtb with the modified one in the flash-rootA:

sudo su && cp imx8mm-var-dart-dt8mcustomboard-legacy.dtb /media/<your_user>/flash-rootA/boot/
sync && unmount /dev/<your_sd_card_partitions>

Then plug in the sd-card and flash the board.

Booting the board on the new image after sd-card provisioning finished will not work because you’re only changing the flasher image device tree, but it’s enough to see If the flashing process works to find out if it’s the pcie differences on the custom carrier board that cause the hang.

Let us know if your carrier board boots the sd-image this way

80

Update: I’ve received confirmation from Variscite that carrier boards that do not have PCIE oscillator should have the pcie node disabled in their device-tree to avoid the kernel hang, and this applies to all custom carrier boards. Also, carrier boards based on DT8M-Customboard versions prior to 1.3 are not supported in kernel 5.4.

So let us know if the steps I described above solves the boot issue on your side.

81

Ok that sounds much more promising. Our custom carrier does NOT have PCIE so that seems like a likely problem. Will try to test this out tonight.

Assuming this works, is the best practice for modifying the dtb as you described?

83

We’ll need to look into adding a possibility of switching the current used dtb in the new OS image if this is the case. Did you manage to test with the above steps?

86

Hello @anathan84

Did you have any luck with the suggestions made by my colleague?

88

@markcorbinuk @acostach since there wasn’t a clear way for us to do the dtb modification in production, we focused our efforts on mitigating the issues we saw using changes in our application code. This is obviously suboptimal and not the route we were hoping for.

Are there any updates on how we can possibly get this into an image that we can OTA update to our existing fleet of 50 devices?

Thanks

Aaron

89

Hi Aaron,

I see. Were you able to confirm that the modified dtb that has pcie disabled solves the issue on a test device, and that the lack of pcie oscillator on the board is the root cause?