Stuck on "Moving /dev, /proc and /sys onto rootfs..."

I have ran “balenaOS 2.58.0” on STM32MP1 for a while by building it with barys-script in deveoper mode.
However, when I build the image with just the machine argument the image will freeze at Moving /dev, /proc and /sys onto rootfs…

What is wrong or how do I troubleshoot it as I can’t interact via console or balena cloud?

Starting kernel …

Uncompressing Linux... done, booting the kernel.
[    1.065129] mmc0: unrecognised SCR structure version 4
[    1.068810] mmc0: error -22 whilst initialising SD card
DEBUG: Loading module debug
DEBUG: Running debug_run
DEBUG: Calling module hook (post): debug_hook_handler
DEBUG: Finished module hook (post): debug_hook_handler
DEBUG: Loading module udev
DEBUG: Calling module hook (pre): debug_hook_handler
DEBUG: Finished module hook (pre): debug_hook_handler
DEBUG: Running udev_run
Unknown filesystem type 62656572 mounted on /sys/fs/cgroup.
Failed to determine root cgroup, ignoring cgroup memory limit: No medium found
Unknown filesystem type 62656572 mounted on /sys/fs/cgroup.
starting version 239
device-enumerator: scan all dirs
  device-enumerator: scanning /sys/bus
  device-enumerator: scanning /sys/class
DEBUG: Calling module hook (post): debug_hook_handler
DEBUG: Finished module hook (post): debug_hook_handler
DEBUG: Loading module prepare
DEBUG: Calling module hook (pre): debug_hook_handler
DEBUG: Finished module hook (pre): debug_hook_handler
DEBUG: Calling module hook (pre): udev_shutdown_hook_handler
DEBUG: Finished module hook (pre): udev_shutdown_hook_handler
DEBUG: Running prepare_run
DEBUG: Calling module hook (post): debug_hook_handler
DEBUG: Finished module hook (post): debug_hook_handler
DEBUG: Loading module fsuuidsinit
DEBUG: Calling module hook (pre): debug_hook_handler
DEBUG: Finished module hook (pre): debug_hook_handler
DEBUG: Calling module hook (pre): udev_shutdown_hook_handler
DEBUG: Finished module hook (pre): udev_shutdown_hook_handler
DEBUG: Skipping module fsuuidsinit
DEBUG: Loading module fsck
DEBUG: Calling module hook (pre): debug_hook_handler
DEBUG: Finished module hook (pre): debug_hook_handler
DEBUG: Calling module hook (pre): udev_shutdown_hook_handler
DEBUG: Finished module hook (pre): udev_shutdown_hook_handler
DEBUG: Running fsck_run
DEBUG: Calling module hook (post): debug_hook_handler
DEBUG: Finished module hook (post): debug_hook_handler
DEBUG: Loading module resindataexpander
DEBUG: Calling module hook (pre): debug_hook_handler
DEBUG: Finished module hook (pre): debug_hook_handler
DEBUG: Calling module hook (pre): udev_shutdown_hook_handler
DEBUG: Finished module hook (pre): udev_shutdown_hook_handler
DEBUG: Running resindataexpander_run
DEBUG: Calling module hook (post): debug_hook_handler
DEBUG: Finished module hook (post): debug_hook_handler
DEBUG: Loading module rorootfs
DEBUG: Calling module hook (pre): debug_hook_handler
DEBUG: Finished module hook (pre): debug_hook_handler
DEBUG: Calling module hook (pre): udev_shutdown_hook_handler
DEBUG: Finished module hook (pre): udev_shutdown_hook_handler
DEBUG: Skipping module rorootfs
DEBUG: Loading module rootfs
DEBUG: Calling module hook (pre): debug_hook_handler
DEBUG: Finished module hook (pre): debug_hook_handler
DEBUG: Calling module hook (pre): udev_shutdown_hook_handler
DEBUG: Finished module hook (pre): udev_shutdown_hook_handler
DEBUG: Running rootfs_run
DEBUG: No e2fs compatible filesystem has been mounted, mounting PARTUUID=3b807b18-a778-408a-bfb8-12ca22a3a220...
DEBUG: Calling module hook (post): debug_hook_handler
DEBUG: Finished module hook (post): debug_hook_handler
DEBUG: Loading module machineid
DEBUG: Calling module hook (pre): debug_hook_handler
DEBUG: Finished module hook (pre): debug_hook_handler
DEBUG: Calling module hook (pre): udev_shutdown_hook_handler
DEBUG: Finished module hook (pre): udev_shutdown_hook_handler
DEBUG: Running machineid_run
DEBUG: Calling module hook (post): debug_hook_handler
DEBUG: Finished module hook (post): debug_hook_handler
DEBUG: Loading module finish
DEBUG: Calling module hook (pre): debug_hook_handler
DEBUG: Finished module hook (pre): debug_hook_handler
DEBUG: Calling module hook (pre): udev_shutdown_hook_handler
DEBUG: Finished module hook (pre): udev_shutdown_hook_handler
DEBUG: Running finish_run
DEBUG: Moving /dev, /proc and /sys onto rootfs...

Hi, I’m reading through the logs to see whats happening. For interacting with the device it would be good to know if is this a remote device or if you have access to it?

I have access to the device so I can take the SD-card and connect it to my computer and check for log-files.

So where do I find those logs?

Hello Anders, could you please go to the balenaCloud and click on the arrow next to the restart button, then click Grant support access so we can access to the device and see what happens? Thank you!

Done

By the way. I have copied config.json from sd-card running balenaOS in dev mode to this sd-card running in production mode. That is why there is also older logs.

could you please share the URL or the ID of your device?

UUID: 63c685c6693174904b72af1b2a2dc8a0

Hi,

I can see it was fixed now, but what was the problem?

I can now see that I missunderstod how the balenaOS console would react when running in production mode. I was waiting for the login prompt.
However, my problem was that the container never was started. Why didn’t it do that?
Was it that I copied config.json from a card running in developer mode and balena cloud didn’t understand that this card didn’t have any container downloaded?

HI Anders, you have a good description of development vs production images in https://www.balena.io/docs/reference/OS/overview/2.x/#development-vs-production-images. As you have found out, the console is not enabled in production images.
The information about current installed containers is stored in the data partition, so they only way it could be lost if if you have reprogrammed BalenaOS in the storage media. A hostOS update triggered via the dashboard will not remove the container information.

That does not explain why my device didn’t want to download my container as I ran a new clean balenaOS with just config.json copied from my old one.
I was thinking that you fixed it for me, but maybe it was only me changing settings in balenaCloud that got it running. I think I changed pin to release.

Just to elaborate on the issue -

  • you built a prod. version of balenaOS for your device, copied across a config.json from an SD card that was being used to run a dev version of balenaOS
  • your prod. device appeared in the dashboard, but the service containers were not being downloaded - however that was fixed when you changed pin to release

Is this correct?

Yes, I think that is what I did if your team didn’t help me as I Grant support access

Hello,
At the moment, we heard back from a few people who worked on your issue but they didn’t fix the device on their end. We usually inform customers of any solutions or fixes we make on the device whenever possible to keep them updated.

Regarding copying the config.json, I would advise instead of doing that you can your to configure your unmanaged balenaOS by the balena os configure command. Hope it helps! Reference - https://www.balena.io/docs/reference/balena-cli/#os-configure-image

I don’t think this is a problem anymore as it work if I do the right way and not copy config.json from a managed device to a unmanaged image.

balena os configure would be great way to do it if it only worked. My STM32MP1 image file do have 2 more partitions included that I guess break how the balena command work.
I get Unsupported filesystem error.

My Balena image file contains:

fsbl1.img - Linux Data
fsbl2.img - Linux Data
resin-boot.img - EFI System
resin-data.img - Linux Data
resin-rootA.img - Linux Data
resin-rootB.img - Linux Data
resin-state.img - Linux Data
ssbl.img - Linux Data