Preloaded containers doesn't start on Intel NUC 2.50.1

Hi,

In the Intel NUC 2.50.1 (latest release), preloaded containers doesn’t start any more. Only supervisor is running. I can list all the preloaded images on the device itself (balena image ls)

But it works with for example 2.48 version.

Any changes made ?

Regards,
Raghav

Hi, can you send us logs from journalctl -a -l --no-pager command?

@karaxuna Please find the logs attached. Thanks.journalctl.log (148.8 KB)

Hey, could you please give us some extra context about what exactly are you trying to do?
Those errors look like somebody trying to use sshd incorrectly.
Jul 02 10:45:34 balena sshd[1706]: error: Could not load host key: /etc/ssh/hostkeys/ssh_host_dsa_key
Thanks!

@georgiats Sure! I am currently testing on the development version of the BalenaOS which does not require ssh keys pre-configured. Hence the error and I think it is not important.

I could reproduce the actual issue on both production & development images for the latest version of Intel NUC 2.50.1 BalenaOS.

The issue is simple, we preload certain containers from a release using the balena cli to the fresh Intel NUC BalenaOS.

From the documentation , For example :
$ balena preload balena.img --app 1234 --commit e1f2592fc6ee949e68756d4f4a48e49bff8d72a0 --splash-image image.png

It works perfectly well till BalenaOS 2.48 x but with the latest Balena OS, the preloaded containers won’t start anymore. Please let me know if you need more information.

Regards,
Raghav

Hello, could you send us specifically the supervisor logs to have a closer look? (journalctl -u resin-supervisor --no-pager -n 200). This is strange as I can see in the logs above the supervisor detecting the preloaded application and claiming it to have started it up successfully. Just to be sure: are you testing the same release/commit across different OS versions, and did you also observe that release running fine when not preloaded?

@nazrhom Please find the logs attached.

Yes, I am testing the same release on both the versions of the OS. 2.48 works just fine. The issue is with the latest 2.50.X version. It’s the same behaviour for both dev and prod versions as well.

It’s fairly easy to reproduce !

supervisor.log (8.3 KB)

journalctl-supervisor.log (3.9 KB)

I’ve tried to reproduce this on a pi4, and the preloading worked perfectly. I’ll dig out an intel nuc now and have another go. If I can’t reproduce, perhaps something is going wrong in relation to your release, although that seems very unlikely.

I tried this with an intel nuc too, and everything worked fine for me. Have you tried all the standard stuff like redownloading your image, using a different usb key etc?

If so, could you post an exact step-by-step guide of what you’re doing? It could be that my pathway is slightly different to yours, and that’s whats causing the inconsistency.

@CameronDiver Thanks for your effort. And sorry for late replay as I was busy working on a workshop.

Here are the steps that I follow :

Step 1: Download latest Intel NUC production image from the BalenaOS website
https://api.balena-cloud.com/download?deviceType=intel-nuc&version=2.50.1+rev1.prod&fileType=.zip

Step 2: Preload load the Containers from a particular release id from an application.

balena preload balena-cloud-intel-nuc-2.50.1+rev1-dev-v11.4.10.img --app TestAPP --commit e1f2592fc6ee949e68756d4f4a48e49bff8d72a0 --splash-image logo.png

Step 3: I use resin-image-flasher-unwrap tool to get the embedded balena image from balena image flasher in the VMDK format so that I can run in the VMware Hypervisor

./docker-run /complete-path/balena-cloud-intel-nuc-2.50.1+rev1-dev-v11.4.10.img --format vmdk

Step 4: Run the VMDK in the vmware hypervisor.

With all the above steps, the preloaded containers in BalenaOS 2.48 version works well. The issue starts from the image 2.50+

Hi,
Could you please clarify if the device you try this on has Internet connectivity?

Hi @roman-mazur , Yes it does ! Is internet connection required to run pre-loaded containers ?

Regards,
Raghav

Hey Raghav, I’m going to try out the process you provided that uses VMDK steps since other have tried this and it works for them and only difference is they aren’t using VMDK. Can you confirm that you are also experiencing the issue without using VMDK ? Also let me know what the need is for running balenaOS in a VM. Thanks!

Hey Raghav, I’ve finally had a chance to follow the steps you provided which should have shown my preloaded containers not running. When I followed the steps except using Virtualbox rather then VMware and didn’t set a splash-image, my preloaded container was running. I used Intel Nuc balenaOS 2.50.1+rev1 with Supervisor 11.4.10.

Can you try to reproduce again with your steps and also use another project like our sample hello worlds. Here is a Go version which will build quickly https://github.com/balena-io-examples/balena-go-hello-world.

I will note though that I could forsee so many subtle differences from running balenaOS on the host vs in a VM that could add lots of complexity when troubleshooting issues. I ask again, let us know why you want to run balenaOS in a VM as I think you should run it directly on the NUC.

@20k-ultra Hi Miguel,

I tried like you suggested with the balena-go-hello-world and it still is not working. I tried on both Intel NUC physical device (UP board) and in a VM without any success. What could be going wrong ? The preloaded image is on the device (I can confirm it with balena image ls command) But just Supervisor container starts and not the other preloaded container.

Again, when I follow the same steps for 2.48 Balena OS versions it works fine ! I updated the balena-cli to the latest version as well. What are the other variables that might be the cause ?

We maintain both the variants of the images i.e both for physical devices and virtual machines. It just developed because of some customer requirements in their enterprise environment.

Regards,
Raghav

Thanks for trying that out Raghav. Since we are unable to reproduce the issue it’ll be a bit more challenging but we’ll definitely keep trying to figure it out. Could you do the same steps that creates the issue but at the end when you see that your container is not running can you go to your dashboard -> application -> select device -> Diagnostics tab -> select Device Diagnostics -> cick Run diagnostics and share the log file generated please.

https://www.balena.io/docs/learn/more/masterclasses/device-debugging/#22-device-diagnostics