Hello,
I am looking to run a virtual machine image inside of a container for a specific use case. I have been able to do this using QEMU inside of a Balena container. However, I am looking to virtualize using KVM hypervisor instead of QEMU.
That said, does BalenaOS support KVM and if so how would I go about enabling that feature. Can this be enabled through ENV variable in the compose file being used for Balena containers?
I have been able to achieve this through basic docker on a linux box. Just wondering how to best achieve this with BalenaCloud.
Hi @jockdarock . I don’t think we or any other customers have ever tried to run KVM emulation in a container, so it’s not an officially supported feature. That being said, I can’t see why it shouldn’t work, as long as the device type you are using has all the correct kernel modules enabled (no idea what it requires of the host system). You should then just be able to craft a docker-compose file with the same “docker run” commands you used on your basic docker + linux box setup.
I was pretty sure on the host side kvm would also have to be installed. And I think I have had trouble with it before in balena, but it has been 2-3 weeks since I have tried it so I am probably forgetting something. I will try it again though and see what happens.
I’ve got a similar use case… We’ve got virtualisation working in a container running on Ubuntu server and it’s working with a simple Dockerfile (Installs qemu-kvm package then runs kvm).
I’m in the process of porting across to evaluate if BalenaOS can cope with this, but when I run kvm I get the following:
Could not access KVM kernel module: No such file or directory qemu-system-x86_64: failed to initialize KVM: No such file or directory
If I look for the module, I suspect our problem is with BalenaOS not supporting it:
root@1bfe64c4dad6:/# modprobe kvm-intel modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.2.10-yocto-standard/modules.dep.bin' modprobe: FATAL: Module kvm-intel not found in directory /lib/modules/5.2.10-yocto-standard
Can anyone offer any advice as to how I can bring KVM to life?
Hi @gk1, as you point out this isn’t supported as the CONFIG_KVM setting is not enabled in the kernel. So you would need to build an image with the CONFIG_KVM setting.
Hello, that does look correct, did you manage to get everything working?
Just in case, once you have the image build and booted, you can check if this has been correctly enabled in the kernel config by doing:
Not quite there… Looks like or corporate firewall blocked some downloads (git:// prefix).
I’ve added some local gitconfig to rewrite these to be https://, and reran the fina barys command, but met the following errors again:
ERROR: resin-image-flasher-1.0-r0 do_rootfs: /home/user/balena-intel/build/tmp/deploy/images/genericx86-64/grub-efi-bootx64.efi is an invalid path referenced in RESIN_BOOT_PARTITION_FILES.
ERROR: resin-image-flasher-1.0-r0 do_rootfs: Function failed: resin_boot_dirgen_and_deploy (log file is located at /home/user/balena-intel/build/tmp/work/genericx86_64-poky-linux/resin-image-flasher/1.0-r0/temp/log.do_rootfs.3729)
ERROR: Logfile of failure stored in: /home/user/balena-intel/build/tmp/work/genericx86_64-poky-linux/resin-image-flasher/1.0-r0/temp/log.do_rootfs.3729
ERROR: Task (/home/user/balena-intel/build/../layers/meta-balena/meta-balena-common/recipes-core/images/resin-image-flasher.bb:do_rootfs) failed with exit code '1'
ERROR: resin-image-1.0-r0 do_rootfs: /home/user/balena-intel/build/tmp/deploy/images/genericx86-64/grub-efi-bootx64.efi is an invalid path referenced in RESIN_BOOT_PARTITION_FILES.
ERROR: resin-image-1.0-r0 do_rootfs: Function failed: resin_boot_dirgen_and_deploy (log file is located at /home/user/balena-intel/build/tmp/work/genericx86_64-poky-linux/resin-image/1.0-r0/temp/log.do_rootfs.3721)
ERROR: Logfile of failure stored in: /home/user/balena-intel/build/tmp/work/genericx86_64-poky-linux/resin-image/1.0-r0/temp/log.do_rootfs.3721
ERROR: Task (/home/user/balena-intel/build/../layers/meta-balena/meta-balena-common/recipes-core/images/resin-image.bb:do_rootfs) failed with exit code '1'
Hi,
The grub-efi-bootx64.efi image is created by the grub-efi package, placed into the deploy folder and copied into the boot partition of the image so it can be ready at boot time.
The error message above just says that the artifact was not found on the deploy folder so it can’t be copied to the image.
Please try the following from the balena-intel repository root and send the log back:
Getting closer now!!! ssh’ing to main balena OS I see the following:
root@balena:~# zcat /proc/config.gz |grep -i kvm
CONFIG_KVM_GUEST=y
# CONFIG_KVM_DEBUG_FS is not set
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_HAVE_KVM_IRQFD=y
CONFIG_HAVE_KVM_IRQ_ROUTING=y
CONFIG_HAVE_KVM_EVENTFD=y
CONFIG_KVM_MMIO=y
CONFIG_KVM_ASYNC_PF=y
CONFIG_HAVE_KVM_MSI=y
CONFIG_HAVE_KVM_CPU_RELAX_INTERCEPT=y
CONFIG_KVM_VFIO=y
CONFIG_KVM_GENERIC_DIRTYLOG_READ_PROTECT=y
CONFIG_KVM_COMPAT=y
CONFIG_HAVE_KVM_IRQ_BYPASS=y
CONFIG_KVM=m
# CONFIG_KVM_INTEL is not set
# CONFIG_KVM_AMD is not set
# CONFIG_KVM_MMU_AUDIT is not set
CONFIG_PTP_1588_CLOCK_KVM=y
From inside my container, if I run /usr/bin/kvm I still get the following:
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
I have io.balena.features.kernel-modules: '1' in my docker-compose file…
Some more info to show the current situation inside my container:
root@2596afe9e233:/# find /lib/modules/ |grep kvm
/lib/modules/5.2.10-yocto-standard/kernel/arch/x86/kvm
/lib/modules/5.2.10-yocto-standard/kernel/arch/x86/kvm/kvm.ko
root@2596afe9e233:/# modprobe kvm
root@2596afe9e233:/# lsmod |grep kvm
kvm 552960 0
irqbypass 16384 1 kvm
root@2596afe9e233:/# /usr/bin/kvm
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: failed to initialize KVM: No such file or directory
Hi,
I wonder if kvm is trying to access a different kernel module (i.e. kvm-intel and not kvm)… Not sure what the difference is - just the best guess.
