Variscite VAR-SOM-MX8M-MINI Boot Freeze

Hello,

We are using the Variscite VAR-SOM-MX8M-MINI starter kit board. I flashed the Variscite DART-MX8M Mini version BalenaOS 2.85.2+rev1. I was assuming (maybe I should have not :smiley: ) that this should work for the VAR-SOM-MX8M-MINI board since Varisicite uses the same DART image for this board but with different DTB files that are selected by UBoot.

I successfully flashed the image and removed the SD card but for some reason, the bootup process freezes during the Bluetooth init (see attached log). I was hoping for some help in narrowing down the issue. We are wanting to use this product for our fleet but need to make it work first :slight_smile:.

Balenos_Boot_Freeze.txt (30.5 KB)

I also tried to compile the latest image from the Github repo b0ebe67e8e84410d6b5be9f2d90befa2c7b96aba. I flashed the SD card with the image balena-image-imx8mm-var-dart.balenaos-img. The bootup freezes in different place.

Last_Balena_Image.txt (26.9 KB)

I noticed that the kernel version is different than what variscite is using 5.4.85 vs 5.4.148. Both freezes are happening after the message Welcome to balenaOS 2.88.4.

Follow up on this activity from me. I am able to see the device in the balenaCloud and I am able to locally SSH into the unit.

I am confused though because the serial terminal output does not print anything else after this statement [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Is the serial terminal disabled for this image in development mode? I am new to this product so I am not sure of the expected behavior. Any help would be appreciated.

Hi, I would recommend you build a custom image and define the OS_DEVELOPMENT variable in conf/local.conf if you are planning to do BSP development.
The available development images should provide a serial login after boot, but given that you are working with an unsupported hardware it may be that’s not the case. Did you modify the build so that the correct kernel device tree is used when booting?

Hello Alexgg,

Thank you for your response. The correct DTB file is being loaded imx8mm-var-som-symphony.dtb. Variscite uses the same Yocto build for the DART-MX8M-MINI as the VAR-SOM-MX8M-MINI I am using. The only difference is a different DTB file is loaded during the Uboot process.

I also confirmed when I build the image off of the Balena git repo the development image was enabled in the local.conf.

# Set this to 1 to disable quiet boot and allow bootloader shell access
OS_DEVELOPMENT = "1"

Not sure, what is causing the hang during boot though. Does the unit need to be registered with the cloud to continue even if the config.json has local and development modes enabled?

Hi, could you check whether there is network activity, like ARP or DHCP requests? It may be that the serial log is not appearing but there is an active network connection. If you spoof the DHCP request you can try to ssh into the device using the balena CLI balena ssh <ip> or directly with ssh -p 22222 root@<ip>.
If there is no network activity, is the device rebooting after some time? I would expect the watchdog to be triggering if it had freezed.

It does not reset at all (left it for like 30 minutes so no reset). These are the sequence of events I did.

  1. Flashed the internal storage with image using balena-image-flasher-imx8mm-var-dart.balenaos-img
    Balena_Image_Flash.txt (34.7 KB)

  2. Removed SD Card and switched to boot from INT storage
    Balenos_Boot_Freeze.txt (26.6 KB)

  3. Using sudo balena scan, found ip address and unit information

Scanning for local balenaOS devices... Reporting scan results
- 
  host:          345c64a.local
  address:       192.168.1.89
  osVariant:     development
  dockerInfo: 
    Containers:        1
    ContainersRunning: 1
    ContainersPaused:  0
    ContainersStopped: 0
    Images:            2
    Driver:            overlay2
    SystemTime:        2022-04-14T15:01:11.249923125Z
    KernelVersion:     5.4.85
    OperatingSystem:   balenaOS 2.88.4+rev7
    Architecture:      aarch64
  dockerVersion: 
    Version:    19.03.30
    ApiVersion: 1.40
  1. Using balena ssh <ip_address>, I was able to SSH into the unit directly. I also yesterday was able to get a container to execute on the unit. Everything points that this is working but the serial terminal just freezes as shown in Balenos_Boot_Freeze log. Never reaches the final terminal step. Is there any commands I can execute that will give a clue as to why?

Hi, could you please post the output of the following:

  • systemctl --failed
  • systemctl list-units | grep serial
  • systemctl status serial-getty@serial0.service
  • ps ax | grep getty
  • ls /dev/serial0 -al

Also please check you are using a serial terminal that disables hardware flow control and is configured for 115200 8N1.

Also @wes8ty , looking at imx8mm_var_dart: Set console environment variable in the default envi… · varigit/uboot-imx@a7f6308 · GitHub I suspect a cat /proc/cmdline will show you console=ttymxc3,115200 on the imx8mm-var-som.

If that’s the case you would have to set the serial console to ttymxc3 for the SOM here balena-variscite-mx8/layer.conf at master · balena-os/balena-variscite-mx8 · GitHub because on the DART ttymxc0 is the serial console. The device is probably not freezing during boot in this case, it’s just not showing the login prompt on the expected console on the development image on the SOM.

Hello acostash and alexgg,

That worked! Changing the ttymsc0 → ttymsc3 in the layer.conf. I get the login screen on my serial terminal and everything prints out.

Is there any plan on supporting the VAR-SOM-MX8M-MINI since it uses the same image / yocto receipts as the DART-MX8M-MINI? From my testing, everything else seems to be working and I was able to run a container.

Thanks so much for the help! I knew there were smarter people out there than me that has been stumbling on this issue for almost a week :smiley:

@wes8ty,

I’m glad to hear that worked for you. :slight_smile:

We have an internal call every few weeks about board support, so I’ll bring your request up in the next meeting (April 20th). Is there a commercial need for it by chance? If so, can you share (here or via email) more details about the use-case, quantities, and timeline that might help with our determining the amount of work and prioritization?

Thanks,
Kenna

There is a commercial use for this (although we will have a custom carrier board for this target hardware but the first step always is to get it working on an evaluation board). I have contacted sales already in regards to this.

Hi, I am having the same issue with my VAR-SOM-MX8M-MINI board. There is definitely a commercial interest for me, since I am planning to use BalenaOS on our MX8M boards. Is there any news on support for these boards?

Also @wes8ty can you perhaps share how you got this to work? Looks like I have to follow these steps to get a DTB file and then follow this guide?

Hi @Cybex , I’m planning to release the VAR SOM MX8M Mini Devkit image in the cloud in the next days, I’ll get back to you here once the image becomes available.

That’s wonderful news @acostach, thank you very much. I’ll be waiting for it :wink:

@Cybex the devkit image is now available in balena-cloud.

Thanks you so much! Going to test it today!

Just tested, it works! Thank you so much @acostach !

1 Like