New installed device registers but does not boot

Hi!

I’ve created a balenaCloud app, and successfully flashed an Intel NUC device. It runs my docker-compose.yml based app flawlessly, as far as I can tell. Great!

I’ve just flashed a new PC which is an Axiomtech ICO300, a fanless Intel Atom based industrial PC. It shows up in the balenaCloud interface as a new device, with “Post provisioning” as the final status. After the device powers off, I yank the USB flash drive, and reboot. On the attached screen, I get this error message (hand-typed, apologies for any errors):

CP437: Invalid argument
fsck.fat 4.1 (2017-01-24)
open: No such file or directory
fsck.ext4: No such file or directory while trying to open /dev/disk/by-label/resin-rootA
Possibly non-existent device?
fsck.ext4: No such file or directory while trying to open /dev/disk/by-label/resin-rootB
Possibly non-existent device?
fsck.ext4: No such file or directory while trying to open /dev/disk/by-label/resin-state
Possibly non-existent device?
Warning: Unable to open /dev read-write (Is a directory). /dev has been opened read-only.
Error: /dev: unrecognized disk label
root `/dev/disk/by-label/resin-rootA` doesn't exist or does not contain a /dev

Has anyone encountered this, and can give any advice on how to proceed? I tried the manualfallbackA and -B options in GRUB, but the first (A) results in the same error and manualfallbackB results in “bzImage not found” or something such.

Cheers

Huh. When I open the USB flash drive again on Windows, it has a large number of partitions. Could it be that the balena installer has flashed the USB drive itself?

Hi, that would be my gut feeling as well. It is possible that the installer/flasher process didn’t see the internal drive when it flashed, and somehow picked the USB drive. Could you try writing the USB drive again, and confirm the partition layout, then retry flashing the ICO300 with it and see if that layout changes.

I did it the other way around, booted the PC / ICO300 from an ubuntu stick. the partitions are in fact there, so it must be something wrong with the boot setup?

Hi @pletnes

I seems like the flasher image is registering ok but the flashed OS is struggling to find the partitions. For reference, this is the partition layout [1]

Couple of things will help us in debugging this further.

Can you please list the partitions you see on the usb drive after flashing (if you haven’t over-written the usb with the Ubuntu live image)?

When running from a Ubuntu stick, can you please send the list of partitions on the internal disk?

Also, which OS version are you trying to install?

Thanks

Regards,
ZubairLK

[1] https://www.balena.io/docs/reference/OS/overview/2.x/#image-partition-layout

The disk partitions from your link [1] match exactly what I see on the OS disk, /dev/sda{1..6}. Both the number, ordering and the size. Labels in /dev/disk/by-label/ match too: /dev/disk/by-label/resin-{boot,data,rootA,rootB,state} are all there and are symlinks to /dev/sda{1..6}.

So how do I debug the boot procedure? It’s fsck.ext4 which crashes, can this be disabled like in /etc/fstab (or similar) in an old school linux distro?

(I did not check the USB stick partitions, yet)

Hi. You could add shell-debug in grub to the kernel cmdline so we can have a better idea on what is failing

I tried flashing a different PC of the same model (axiomtek ICO 300) with a different USB stick with a different OS version (2.31.2) and I get the same issue. I noticed briefly in the output on the screen during flashing the error message /dev: unrecognized disk label. So I’m guessing that there’s an error in the flashing procedure itself.

After flashing the PC, the USB stick has the following partitions.

Is this the correct partition layout for the USB flasher? It looks unchanged to me.

For reference: before flashing the PC the USB stick looks like this.

Hi. Can you also try with shell-debug like I suggested in the previous reply?

Sure, where do I put the string shell-debug? Sorry, my linux installs never really had trouble booting, so I didn’t learn the quirks of GRUB2 in any detail…

Hi,
the shell debug belongs in the linux line in the grub configuration.
On an EFI system you would find grub.cfg in:

cat /boot/EFI//BOOT/grub.cfg 
# Automatically created by OE
serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
default=boot
timeout=10

menuentry 'boot'{
linux /bzImage LABEL=boot root=/dev/sda2 

}

So you could try just adding it to the linux line.
On a non EFI sytem it would probably be in /boot/grub

The EFI / boot directory resides on the first partition of the machines drive.

Regards
Thomas

See image. There’s the log from putting shell-debug on the kernel line.

I booted the PC from a Lubuntu stick and I can confirm that the partition layout and names seem sensible. I notice that /dev/sda1 is the resin-boot partition and it has a grub.cfg that looks sensible to the untrained eye, at least.

Thanks for the info, passing it on to our devices team for feedback, and will keep you posted!

What kind of internal storage does this hardware have?

This is the device type. AFAIK it’s a 120/250 GB SSD in this device (I’ve got 2 of them right now). I will boot it from a live usb stick and get back to you tomorrow as I’m not in the office right now.

Intel Atom / 1 core / fanless so it’s an embedded system. I’ve used balena on other Axiomtek’s in the past, but with other CPUs. They boot lubuntu live usb fine so I don’t think drivers for linux are an issue.

OK, to triple check, I grabbed my trusty Philips screwdriver and opened the device. It does indeed have exactly one - 1 - SATA drive, SSD, attached to the motherboard.

Ok, thanks. Can you also add a “rootdelay=10” param to the kernel cmdline just as you did with shell-debug?

Also, can you go in BIOS and make sure SATA is set to AHCI mode?