FreeBSD/Bhyve: Integration of BalenaOS within VM

Versions
FreeBSD:
TwinCAT/BSD(FreeBSD): 12.2-RELEASE-p12 FreeBSD 12.2-RELEASE-p12
vm-bhyve: Bhyve virtual machine management v1.4.2 (rev. 104002)
Balena:
Seleced device type: Intel NUC
Selected version: v2.105.16
Selected OS type: balenaOS

Description
I would like to run my BalenaOS “device” within a virtual machine(bhyve) on my existing PLC.
For the configuration of bhyve I use vm-bhyve.
At startup I have to configure grub to get it started.

I did following configuration:

grub> ls
(cd0) (cd0,msdos6) (cd0,msdos5) (cd0,msdos3) (cd0,msdos2) (cd0,msdos1) (hd0) (h
ost)
grub> ls (cd0,msdos1)/
efi/ balena-image-flasher balenaos.fingerprint config.json device-type.json gru
b/ grub.cfg_internal grub.cfg_internal_luks image-version-info splash/ system-c
onnections/ System Volume Information/
grub> set prefix=(cd0,1)/grub
grub> set root=(hd0,1)
grub> insmod linux
grub> insmod normal
grub> normal

Output:

Linux version 5.14.21-yocto-standard (oe-user@oe-host) (x86_64-poky-linux-gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37.20210721) #1 SMP PREEMPT Mon Nov 29 01:17:48 UTC 2021
Command line: console=ttyS0 BOOT_IMAGE=/boot/bzImage root=LABEL=flash-rootA flasher rootwait
x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
signal: max sigframe size: 1776
BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009fffe] usable
BIOS-e820: [mem 0x0000000000100000-0x000000001fffffff] usable
NX (Execute Disable) protection: active
SMBIOS 2.6 present.
DMI:  BHYVE, BIOS 1.00 03/14/2014
tsc: Fast TSC calibration using PIT
tsc: Detected 3343.546 MHz processor
last_pfn = 0x20000 max_arch_pfn = 0x400000000
Disabled
x86/PAT: MTRRs disabled, skipping PAT initialization too.
CPU MTRRs all blank - virtualized system.
x86/PAT: Configuration [0-7]: WB  WT  UC- UC  WB  WT  UC- UC  
found SMP MP-table at [mem 0x000f0000-0x000f000f]
Using GB pages for direct mapping
ACPI: Early table checksum verification disabled
ACPI: RSDP 0x00000000000F2400 000024 (v02 BHYVE )
ACPI: XSDT 0x00000000000F2480 000044 (v01 BHYVE  BVXSDT   00000001 INTL 20200430)
ACPI: APIC 0x00000000000F2500 00005A (v01 BHYVE  BVMADT   00000001 INTL 20200430)
ACPI: FACP 0x00000000000F25D2 00010C (v05 BHYVE  BVFACP   00000001 INTL 20200430)
ACPI: DSDT 0x00000000000F2852 00097B (v02 BHYVE  BVDSDT   00000001 INTL 20200430)
ACPI: FACS 0x00000000000F2792 000040
ACPI: FACS 0x00000000000F2792 000040
ACPI: HPET 0x00000000000F2712 000038 (v01 BHYVE  BVHPET   00000001 INTL 20200430)
ACPI: MCFG 0x00000000000F2752 00003C (v01 BHYVE  BVMCFG   00000001 INTL 20200430)
ACPI: Reserving APIC table memory at [mem 0xf2500-0xf2559]
ACPI: Reserving FACP table memory at [mem 0xf25d2-0xf26dd]
ACPI: Reserving DSDT table memory at [mem 0xf2852-0xf31cc]
ACPI: Reserving FACS table memory at [mem 0xf2792-0xf27d1]
ACPI: Reserving FACS table memory at [mem 0xf2792-0xf27d1]
ACPI: Reserving HPET table memory at [mem 0xf2712-0xf2749]
ACPI: Reserving MCFG table memory at [mem 0xf2752-0xf278d]
Setting APIC routing to physical flat.
Zone ranges:
  DMA      [mem 0x0000000000001000-0x0000000000ffffff]
  DMA32    [mem 0x0000000001000000-0x000000001fffffff]
  Normal   empty
Movable zone start for each node
Early memory node ranges
  node   0: [mem 0x0000000000001000-0x000000000009efff]
  node   0: [mem 0x0000000000100000-0x000000001fffffff]
Initmem setup node 0 [mem 0x0000000000001000-0x000000001fffffff]
On node 0, zone DMA: 1 pages in unavailable ranges
On node 0, zone DMA: 97 pages in unavailable ranges
ACPI: PM-Timer IO Port: 0x408
ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1])
IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-31
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
ACPI: Using ACPI (MADT) for SMP configuration information
ACPI: HPET id: 0x80860701 base: 0xfed00000
smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[mem 0x20000000-0xffffffff] available for PCI devices
Booting paravirtualized kernel on bare hardware
clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
setup_percpu: NR_CPUS:64 nr_cpumask_bits:64 nr_cpu_ids:1 nr_node_ids:1
percpu: Embedded 57 pages/cpu s195864 r8192 d29416 u2097152
Built 1 zonelists, mobility grouping on.  Total pages: 128768
Kernel command line: console=ttyS0 BOOT_IMAGE=/boot/bzImage root=LABEL=flash-rootA flasher rootwait
Unknown command line parameters: flasher BOOT_IMAGE=/boot/bzImage
Dentry cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
Inode-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 472740K/523896K available (14347K kernel code, 2200K rwdata, 3696K rodata, 14316K init, 1800K bss, 50896K reserved, 0K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
ftrace: allocating 49516 entries in 194 pages
ftrace: allocated 194 pages with 3 groups
rcu: Preemptible hierarchical RCU implementation.
rcu:    RCU event tracing is enabled.
rcu:    RCU restricting CPUs from NR_CPUS=64 to nr_cpu_ids=1.
        Trampoline variant of Tasks RCU enabled.
        Rude variant of Tasks RCU enabled.
        Tracing variant of Tasks RCU enabled.
rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
NR_IRQS: 4352, nr_irqs: 256, preallocated irqs: 16
kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)
random: get_random_bytes called from start_kernel+0x4cc/0x698 with crng_init=0
Console: colour EGA 80x25
printk: console [ttyS0] enabled
ACPI: Core revision 20210604
clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 113919999973 ns
APIC: Switch to symmetric I/O mode setup
..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x3031fc983fd, max_idle_ns: 440795203904 ns
Calibrating delay loop (skipped), value calculated using timer frequency.. 6687.09 BogoMIPS (lpj=3343546)
pid_max: default: 32768 minimum: 301
LSM: Security Framework initializing
landlock: Up and running.
Mount-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes, linear)
Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 512
Last level dTLB entries: 4KB 1536, 2MB 1536, 4MB 768, 1GB 0
Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
Spectre V2 : Mitigation: Full AMD retpoline
Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
Freeing SMP alternatives memory: 20K
smpboot: CPU0: AMD Ryzen Embedded V1807B with Radeon Vega Gfx (family: 0x17, model: 0x11, stepping: 0x0)
Performance Events: PMU not available due to virtualization, using software events only.
rcu: Hierarchical SRCU implementation.
smp: Bringing up secondary CPUs ...
smp: Brought up 1 node, 1 CPU
smpboot: Max logical packages: 1
smpboot: Total of 1 processors activated (6687.09 BogoMIPS)
devtmpfs: initialized
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
futex hash table entries: 256 (order: 2, 16384 bytes, linear)
pinctrl core: initialized pinctrl subsystem
NET: Registered PF_NETLINK/PF_ROUTE protocol family
thermal_sys: Registered thermal governor 'step_wise'
thermal_sys: Registered thermal governor 'user_space'
cpuidle: using governor menu
ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
ACPI: bus type PCI registered
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
PCI: not using MMCONFIG
PCI: Using configuration type 1 for base access
PCI: Using configuration type 1 for extended access
Kprobes globally optimized
raid6: avx2x4   gen() 21363 MB/s
raid6: avx2x4   xor()  8075 MB/s
raid6: avx2x2   gen() 23509 MB/s
raid6: avx2x2   xor() 14568 MB/s
raid6: avx2x1   gen() 19352 MB/s
raid6: avx2x1   xor() 12230 MB/s
raid6: sse2x4   gen() 13230 MB/s
raid6: sse2x4   xor()  7312 MB/s
raid6: sse2x2   gen() 13533 MB/s
raid6: sse2x2   xor()  8719 MB/s
raid6: sse2x1   gen() 11642 MB/s
raid6: sse2x1   xor()  6220 MB/s
raid6: using algorithm avx2x2 gen() 23509 MB/s
raid6: .... xor() 14568 MB/s, rmw enabled
raid6: using avx2x2 recovery algorithm
ACPI: Added _OSI(Module Device)
ACPI: Added _OSI(Processor Device)
ACPI: Added _OSI(3.0 _SCP Extensions)
ACPI: Added _OSI(Processor Aggregator Device)
ACPI: Added _OSI(Linux-Dell-Video)
ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
ACPI: 1 ACPI AML tables successfully acquired and loaded
ACPI: Interpreter enabled
ACPI: PM: (supports S0 S5)
ACPI: Using IOAPIC for interrupt routing
PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000)
PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources
PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00])
acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
pci_bus 0000:00: root bus resource [io  0x0d00-0x1fff window]
pci_bus 0000:00: root bus resource [io  0x2000-0x203f window]
pci_bus 0000:00: root bus resource [mem 0xc0000000-0xc01fffff window]
pci_bus 0000:00: root bus resource [mem 0x800000000-0x8000fffff window]
pci_bus 0000:00: root bus resource [bus 00]
pci 0000:00:00.0: [1275:1275] type 00 class 0x060000
pci 0000:00:03.0: [8086:2821] type 00 class 0x010601
pci 0000:00:03.0: reg 0x24: [mem 0xc0002000-0xc00023ff]
pci 0000:00:04.0: [8086:2821] type 00 class 0x010601
pci 0000:00:04.0: reg 0x24: [mem 0xc0002400-0xc00027ff]
pci 0000:00:05.0: [1af4:1000] type 00 class 0x020000
pci 0000:00:05.0: reg 0x10: [io  0x2000-0x201f]
pci 0000:00:05.0: reg 0x14: [mem 0xc0000000-0xc0001fff]
pci 0000:00:1f.0: [8086:7000] type 00 class 0x060100
ACPI: PCI: Interrupt link LNKA configured for IRQ 5
ACPI: PCI: Interrupt link LNKB configured for IRQ 6
ACPI: PCI: Interrupt link LNKC configured for IRQ 7
ACPI: PCI: Interrupt link LNKD configured for IRQ 0
ACPI: PCI: Interrupt link LNKD disabled
ACPI: PCI: Interrupt link LNKE configured for IRQ 0
ACPI: PCI: Interrupt link LNKE disabled
ACPI: PCI: Interrupt link LNKF configured for IRQ 0
ACPI: PCI: Interrupt link LNKF disabled
ACPI: PCI: Interrupt link LNKG configured for IRQ 0
ACPI: PCI: Interrupt link LNKG disabled
ACPI: PCI: Interrupt link LNKH configured for IRQ 0
ACPI: PCI: Interrupt link LNKH disabled
iommu: Default domain type: Translated 
vgaarb: loaded
SCSI subsystem initialized
ACPI: bus type USB registered
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
PTP clock support registered
Advanced Linux Sound Architecture Driver Initialized.
Bluetooth: Core ver 2.22
NET: Registered PF_BLUETOOTH protocol family
Bluetooth: HCI device and connection manager initialized
Bluetooth: HCI socket layer initialized
Bluetooth: L2CAP socket layer initialized
Bluetooth: SCO socket layer initialized
PCI: Using ACPI for IRQ routing
hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
hpet0: 8 comparators, 32-bit 16.777216 MHz counter
clocksource: Switched to clocksource tsc-early
pnp: PnP ACPI init
system 00:02: [io  0x0220-0x0223] has been reserved
system 00:02: [io  0x0224-0x0227] has been reserved
system 00:02: [io  0x04d0-0x04d1] has been reserved
system 00:02: [io  0x0400-0x0407] has been reserved
system 00:02: [mem 0xe0000000-0xefffffff] has been reserved
pnp: PnP ACPI: found 6 devices
clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
NET: Registered PF_INET protocol family
IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear)
tcp_listen_portaddr_hash hash table entries: 256 (order: 0, 4096 bytes, linear)
TCP established hash table entries: 4096 (order: 3, 32768 bytes, linear)
TCP bind hash table entries: 4096 (order: 4, 65536 bytes, linear)
TCP: Hash tables configured (established 4096 bind 4096)
UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
NET: Registered PF_UNIX/PF_LOCAL protocol family
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
pci_bus 0000:00: resource 5 [io  0x0d00-0x1fff window]
pci_bus 0000:00: resource 6 [io  0x2000-0x203f window]
pci_bus 0000:00: resource 7 [mem 0xc0000000-0xc01fffff window]
pci_bus 0000:00: resource 8 [mem 0x800000000-0x8000fffff window]
pci 0000:00:1f.0: Activating ISA DMA hang workarounds
PCI: CLS 0 bytes, default 64
Initialise system trusted keyrings
workingset: timestamp_bits=46 max_order=17 bucket_order=0
NFS: Registering the id_resolver key type
Key type id_resolver registered
Key type id_legacy registered
Key type cifs.idmap registered
aufs 5.x-rcN-20210308
xor: automatically using best checksumming function   avx       
Key type asymmetric registered
Asymmetric key parser 'x509' registered
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
io scheduler mq-deadline registered
io scheduler kyber registered
input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
ACPI: button: Power Button [PWRF]
virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
Serial: 8250/16550 driver, 32 ports, IRQ sharing disabled
00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
00:04: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
Linux agpgart interface v0.103
brd: module loaded
loop: module loaded
zram: Added device: zram0
hv_vmbus: registering driver hv_storvsc
ahci 0000:00:03.0: SSS flag set, parallel bus scan disabled
ahci 0000:00:03.0: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x1 impl SATA mode
ahci 0000:00:03.0: flags: 64bit ncq sntf ilck stag pm led clo pio slum part sxs apst 
scsi host0: ahci
scsi host1: ahci
scsi host2: ahci
scsi host3: ahci
scsi host4: ahci
scsi host5: ahci
ata1: SATA max UDMA/133 abar m1024@0xc0002000 port 0xc0002100 irq 32
ata2: DUMMY
ata3: DUMMY
ata4: DUMMY
ata5: DUMMY
ata6: DUMMY
ahci 0000:00:04.0: SSS flag set, parallel bus scan disabled
ahci 0000:00:04.0: AHCI 0001.0300 32 slots 6 ports 6 Gbps 0x1 impl SATA mode
ahci 0000:00:04.0: flags: 64bit ncq sntf ilck stag pm led clo pio slum part sxs apst 
scsi host6: ahci
scsi host7: ahci
scsi host8: ahci
scsi host9: ahci
scsi host10: ahci
scsi host11: ahci
ata7: SATA max UDMA/133 abar m1024@0xc0002400 port 0xc0002500 irq 33
ata8: DUMMY
ata9: DUMMY
ata10: DUMMY
ata11: DUMMY
ata12: DUMMY
tun: Universal TUN/TAP device driver, 1.6
e100: Intel(R) PRO/100 Network Driver
e100: Copyright(c) 1999-2006 Intel Corporation
e1000: Intel(R) PRO/1000 Network Driver
e1000: Copyright (c) 1999-2006 Intel Corporation.
e1000e: Intel(R) PRO/1000 Network Driver
e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
ehci-platform: EHCI generic platform driver
ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
ohci-pci: OHCI PCI platform driver
uhci_hcd: USB Universal Host Controller Interface driver
usbcore: registered new interface driver usb-storage
usbcore: registered new interface driver usbserial_generic
usbserial: USB Serial support registered for generic
i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
i8042: Warning: Keylock active
serio: i8042 KBD port at 0x60,0x64 irq 1
serio: i8042 AUX port at 0x60,0x64 irq 12
hv_vmbus: registering driver hyperv_keyboard
mousedev: PS/2 mouse device common for all mice
input: AT Raw Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
rtc_cmos 00:05: registered as rtc0
rtc_cmos 00:05: setting system clock to 2022-10-26T17:04:57 UTC (1666803897)
rtc_cmos 00:05: alarms up to one day, y3k, 114 bytes nvram
fail to initialize ptp_kvm
device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
sdhci-pltfm: SDHCI platform and OF driver helper
hid: raw HID events driver (C) Jiri Kosina
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
u32 classifier
    input device check on
    Actions configured
IPVS: Registered protocols ()
IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
IPVS: ipvs loaded.
NET: Registered PF_INET6 protocol family
Segment Routing with IPv6
sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver
NET: Registered PF_PACKET protocol family
Bridge firewalling registered
Key type dns_resolver registered
NET: Registered PF_VSOCK protocol family
IPI shorthand broadcast: enabled
sched_clock: Marking stable (718483445, 50119207)->(770020676, -1418024)
registered taskstats version 1
Loading compiled-in X.509 certificates
Key type ._fscrypt registered
Key type .fscrypt registered
Key type fscrypt-provisioning registered
Btrfs loaded, crc32c=crc32c-generic, zoned=no
Key type encrypted registered
printk: console [netcon0] enabled
netconsole: network logging started
ALSA device list:
  No soundcards found.
ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata1.00: ATAPI: BHYVE SATA DVD ROM, 001, max MWDMA2
ata1.00: configured for PIO4
ata7: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata7.00: ATA-9: BHYVE SATA DISK, 001, max MWDMA2
ata7.00: 20971520 sectors, multi 0: LBA48 NCQ (depth 32)
ata7.00: configured for PIO4
scsi 0:0:0:0: CD-ROM            BHYVE    BHYVE DVD-ROM    001  PQ: 0 ANSI: 5
sr 0:0:0:0: [sr0] scsi3-mmc drive: 0x/0x xa/form2 caddy
cdrom: Uniform CD-ROM driver Revision: 3.20
scsi 6:0:0:0: Direct-Access     ATA      BHYVE SATA DISK  001  PQ: 0 ANSI: 5
sd 6:0:0:0: [sda] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
sd 6:0:0:0: [sda] 131072-byte physical blocks
sd 6:0:0:0: [sda] Write Protect is off
sd 6:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 6:0:0:0: [sda] Attached SCSI disk
Freeing unused kernel image (initmem) memory: 14316K
Write protecting the kernel read-only data: 20480k
Freeing unused kernel image (text/rodata gap) memory: 2036K
Freeing unused kernel image (rodata/data gap) memory: 400K
Run /init as init process
Starting version 249
virtio_net virtio0 enp0s5: renamed from eth0
random: fast init done
zram0: detected capacity change from 0 to 489776
input: PS/2 Generic Mouse as /devices/platform/i8042/serio1/input/input3
tsc: Refined TSC clocksource calibration: 3341.303 MHz
clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x3029b5af021, max_idle_ns: 440795308761 ns
clocksource: Switched to clocksource tsc
clocksource: timekeeping watchdog on CPU0: Marking clocksource 'tsc' as unstable because the skew is too large:
clocksource:                       'hpet' wd_nsec: 497499644 wd_now: 203914c wd_last: 1843529 mask: ffffffff
clocksource:                       'tsc' cs_nsec: 497858097 cs_now: a6e058e8d61 cs_last: a6da267a35d mask: ffffffffffffffff
clocksource:                       'tsc' is current clocksource.
tsc: Marking TSC unstable due to clocksource watchdog
TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
sched_clock: Marking unstable (2051579037, 50119215)<-(2103117408, -1418024)
clocksource: Not enough CPUs to check clocksource 'tsc'.
clocksource: Switched to clocksource hpet
root '/dev/disk/by-label/flash-rootA' doesn't exist or does not contain a /dev.

So the system is booting, but there is something wrong with my grub configuration?
Do you have a idea what I have to do to get it running?

Thank you!

What I would expect
Working BalenaOS as bhyve guest.

Hi Alexander. Nice to see a TwinCAT / FreeBSD user on here. I don’t know the answer to your question, but did you see this: Root /dev/disk/by-label/flash-rootA doesen't exist or does not contain a /dev · Issue #2027 · balena-os/meta-balena · GitHub ?

@hraftery thank you for your research, but I already saw this and had no success with this information…

So in the meantime I tried to go away of grub and have a try with UEFI.

First I downloaded the Intel NUC image and then my procedure was as follows:

// 1. Create virtual networkadapter

ifconfig tap0 create

sysctl net.link.tap.up_on_open=1

ifconfig bridge0 create

ifconfig bridge0 addm igb0 addm tap0

ifconfig bridge0 up

// 2. Create ZFS volume

sudo zfs create -V10G -o volmode=dev zroot/vms/balenavm

// 3. Copy the balena image to the zfs volume

sudo dd if=Balena.img of=/dev/zvol/zroot/vms/balenavm bs=4M

// 4. Start balena with bhyve

sudo bhyve -AHP -s 0:0,hostbridge -s 1:0,lpc \

> -s 2:0,virtio-net,tap0 -s3:0,virtio-blk,/dev/zvol/zroot/vms/balenavm \

> -c 4 -m 1024M \

> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \

> -l com1,/dev/nmdm0A \

> alpinevm

// 5. Start second terminal and connect to console of balena guest

sudo cu -l /dev/nmdm0B

// 6. VM is running and I can log in with "root"

c059e18 login: root

root@c059e18:~# ls

root@c059e18:~#

So i did one step in the right direction.
But if I look at my balena fleet dashboard I can see now a new added device with the error message “Failed to find any block devices” as status message.

I think It has something to do with the installation instructions… the downloaded image is not the needed rootfs, but only a installation image.?
Do you know how i can get the finished installed image so that i can use it as a hard drive without being able to do the installation process?

So I got it working with this workaround.

  1. Download Intel NUC image

  2. Followed the tutorial how to run balenaos in virtualbox
    => No hardware? Use VirtualBox to simulate a device for balenaCloud!

  3. Converted the resulting *.vdi disc of virtualbox to a *.iso
    => VBoxManage clonehd BalenaVm_1.vdi Balena_ready.iso --format RAW

  4. Used the resulting Balena_ready.iso for my ZFS-Volume / Started VM with bhyve
    => Look at my previous response for the further steps

Now BalenaOS is running fine and is correctly detected in the Balena fleet dashboard.
So the goal is reached but it is a cumbersome procedure…

Do you know if we have here a better procedure to reach my goal?
One additional problem is that I have to do this for each new device again, because if I duplicate the resulting Balena_ready.iso, it would be detected as the same device in the Balena fleet dashboard…