inconsistent device provisioning amd64

Product support
1

Hello,

we are in the process of provisioning 50ish devices. They are all same vendor/model, although from different years so minimal differences in hardware and software versions may occur.

I expected to be able to flash them in bulk, but already with the first 10, sometimes it gets stuck in the postprovisioning step. The process is pretty much the following:

  1. The stick is plugged,
  2. the device is booted from the usb,
  3. a new device appears in balena notifying the progress,
  4. the device switches off and in balena is flagged as “post provisioning”.
  5. I remove the stick and restart the device
  6. the device boots but nothing happens on balena dashbard. The device remains in postprovisioning. Even the device leds, after a few seconds blinking at startup stop doing anything.

After it fails, it doesnt matter how many times I reboot the device, it wont move over the post provisioning status.

If i restart the process from 1, without changing anything on the hardware side, chances are that the device will complete the provisioning.

It is therefore not clear if the issue is on the image, on the bios settings, or something in the communication between the device and balena cloud, however im short of options how to troubleshoot.

Anyone could help?

Btw, this is how the image is built

device_type=generic-amd64
os_version=v6.0.10
image_name=balena-${device_type}-${os_version}.img
fleet_name=<fleet-slug>
app_commit=<release-commit>
balena os download ${device_type} --version ${os_version} --output ${image_name}
balena os configure ${image_name} --version ${os_version} --fleet ${fleet_name} --secureBoot --config-network ethernet
balena preload ${image_name} --fleet ${fleet_name} --commit ${app_commit} --splash-image white-logo.png

and then the image is flashed with balena etcher

2

Hello, How long have you waited with the device powered up after restart and in the post-provisioning state to see if it comes online? Sometimes it can take a while. Does the device show up on your local network that you can ping? Also, if you have a monitor attached, does the balena logo appear briefly at boot?

3

How long have you waited with the device powered up after restart and in the post-provisioning state to see if it comes online?

Admittedly, I didnt wait for too long and will try to be more patient, but as a reference, when everything works fine, the LEDs start blinking hard in less than 10 seconds, but when the issue occurs after a couple of minutes it is still blocked.

Does the device show up on your local network that you can ping?

This I didnt check, but next time it will happen, probably later today, I will check the local network and pinging the device and let you know

if you have a monitor attached, does the balena logo appear briefly at boot?

Yes, the monitor is attached but no logo and no cursor appear, just black screen

4

ok, so it happened again, one device successfully flashed, the second failed. Flashing the second device again also worked smoothly.

Before reflashing it, i did some checks you asked and what I can tell is that when stuck in the postprovisioning, the device is visible in the local network, with the host id assigned as the balena uuid (short), but when trying to ping it, it is not reachable. Other devices are pingable so i would exclude some network setting.

After 10 minutes we proceeded to flash it again and then it worked as expected. How long would you suggest to wait? we can also leave them running overnight it that can help

5

i wanted to follow up in case others should have a similar issue.

This morning I found out that every new player had to be flashed twice with the issue described above. The reason for the first devices to succeed was just because they used a different image.

At one point, when preloading the application I added the --secureBoot, but the device was not configured with secure boot enabled, only UEFI. Since I could not enable setup mode (but this is another story), I gave up with secureBoot for this batch of devices. Will get back to this another time.

Flashing an os image without the --secureBoot param worked smoothly. It would be still interesting to understand the behaviour why flashing two times it worked, but I guess that’s it for now

1 Like
6

Hello @smic thanks for confirming that without the flag --secureBoot

Actually you can read more about configuring UEFIs firmware here if needed BalenaOS Secure Boot & Disk Encryption for x86_64 Devices