Hi, we are at the stage in our product development where we need to lock down the the CM4 to protect company IP. The official RPi usbboot repo has a good tutorial on how to enable and run secure boot.
I’m assuming I’m not the only one that desires to use the secure boot functionality of the CM4 alongside Balena OS so I’m curious if anyone else has already done this and gotten it to work.
our secure boot efforts are currently focused on x86 device types. Once that is finished, we are likely to look into RPi 4 and CM4 but at this moment it is not actively being worked on.
It should be possible to make it work following the turorial though - have you been able to make it work with the minimal boot.img provided as example? If that works for you, could you describe a little more how you generated the boot.img for balenaOS? Are you booting off an SD card or something else?
I’m able to get the minimal boot.img work. I’m booting off the EMMC of a CM4 module.
I might be wrong as I don’t have a lot of experience around how linux boots and loads the kernel but their example loads the FS into RAM and then boots from there.
This is what I’m currently doing.
Mount the resin-boot partition.
Copy contents of that parition to a folder called secure-boot-files
My hunch is that I’m messing something up in this step. I noticed that the Balena cmdline.txt doesn’t have a root specified. But Balena uses a boot script? boot.scr. which pointing it to that might also be my problem.
Set the kernel root device
Since the boot file-system for the firmware is now in a signed disk image the OS cannot write to this. Therefore, any changes to cmdline.txt must be made before the boot.img file is signed.
Verify that cmdline.txt in secure-boot-files points to the correct UUID for the root file-system. Alternatively, for testing, you can specify the root device name e.g. root=/dev/mmcblk0p2.
Remove init-resize.sh from cmdline.txt
Specifically setting the root properly in cmdline.txt
Then I continue on with the tutorial and make the .img
sudo tools/make-boot-image -d secure-boot-files -o boot.img -b cm4 -a 64
thanks for the detailed description, it is perfect. It looks like you did everything correctly, the signature check passes yet it looks like the boot.img is missing some file(s) - the “Firmware not found” message is coming from the bootloader which means balenaOS has not yet started. make-boot-images performs a firmware cleanup by board type, would you be able to inspect your boot.img as described in the tutorial: GitHub - raspberrypi/usbboot: Raspberry Pi USB booting code, moved from tools repository to check which files actually ended up being packaged? Maybe even compare them to the original contents of resin-boot.
The bullets from “Set the kernel root device” do not apply to balenaOS - we do not specify the root device on the command line because the root device is determined dynamically between rootA and rootB. The initrd, which should be packaged with the kernel, takes care of that and at this point I see no obvious reason for it not to work in secure boot mode.
I left it out of what I’ve tried but I did verify the root dir of the image contents matched the resin-boot partition.
Originally it was stripping out the 64 bit kernel because the example was for a Pi4 and it wasn’t obvious that I need to specify -a 64. Once I worked through that problem the contents of the boot.img appeared to match the resin-boot partition.
I will attempt it again and verify the contents more closely. My gut tells me all the files are present but something isn’t pointing it to the correct files.
When it says Firmware not found error 4
Is that a specific file that it can’t find or id it more generically saying 1 or more files in a list of N required files can’t be found?
I verified the contents and discovered that if I leave the -b cm4 in the make-boot-image command then it “works”.
Running with -b cm4 strips the fixup4cd.elf and start4cd.elf. It isn’t clear to me how it selects which fixup*.elf and start*.elf to use but it appears it wanted the cd.* set.
I am now getting significantly further into the boot process now but running into a filesystem error now.
I’m curious if you have any thoughts on this as it appears to be related to the multiple partition structure that Balena OS uses.
DRAM: 7.9 GiB
RPI Compute Module 4 (0xd03140)
MMC: mmcnr@7e300000: 1, mmc@7e340000: 0
Loading Environment from FAT… *** Warning - bad CRC, using default environment
In: serial
Out: serial
Er: serial
Net: eth0: ethernet@7d580000
PCIe BRCM: link down
starting USB…
No working controllers found
Hit any key to stop autoboot: 2 1 0
switch to partitons #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1…
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
7[r[999;999H[6n8Card did not respon to voltage select! : -110
Scanning disk mmcnr@7e300000.blk…
Disk mmcnr@7e300000.blk not ready
Scanning disk mmc@7340000.blk…
** Unrecognized filesystem type **
Found 7 disks
No EFI system partition
BootOrder not defined
EFI boot manager: Cannot load any image
Card did nt respond to voltage select! : -110
starting USB…
No working controllers found
USB is stopped. Please issue ‘usb strt’ first.
starting USB…
No working controllers found
PCIe BRCM: link down
ethernet@7d580000 Waiting for PHY auto negotiation tocomplete… TIMEOUT !
bcmgenet: PHY startup failed: -110
missing environment variable: pxeuuid
missing environment variable: boofile
Retrieving file: pxelinux.cfg/01-e4-5f-01-50-0e-76
from what I can tell, having a plan what implementing a proper Chain of Trust for BalenaOS Secure Boot on the CM4 would be a good start for this project.
As you wrote, your goal is to “lock down the the CM4 to protect company IP.”
IMHO, the word “protect” is not very well defined, and as far as I can tell, it really depends on the class of adversaries you want to protect against.
If your application supports network communication over accessible networks, of course a threat analysis for such protection might also have to include the security of the network communication of the devices as well as all the possible communication endpoints (as well as middle mans, e.g. for MITM attacks). E.g. if your devices talk to Balena Cloud or your own OpenBalena server(s), then also the security of these need to be considered in a threat model.
I am going to write more followup answers, I just wanted to get started from the root.
The Raspberry Pi CM4 boot-loader have to be correctly signed and it is verified on boot.
Since the CM4 device tree and CM4 device tree overlay files are inside the boot.img, these are part of the signed an verified image as well.
As part of the signed boot.img using the file kernel8.img, the U-Boot for BalenaOS is loaded by the CM4 boot loader is signed and verified on boot. Normally, kernel8.img contains the kernel, but in BalenaOS, it contains U-Boot. (The macro resin_set_kernel_root which is built into the BalenaOS U-Boot appears to select the partition on which loading and booting the BalenaOS Kernel happens)
Because the boot ROM of the BCM2711 will only boot a signed bootloader, you also get this:
Prevention of unauthorized modifications of the Boot EEPROM (early boot loader, boot order and secure boot config) of the CM4
Prevention of unauthorized modifications of the eMMC over USB using rpiboot mode (by placing the CM4 into USB gadget mode and then sending arbitrary code over USB)
Especially the last two give you protection against simple reverse-engineering and hacking by someone with physical access to a production device by using the USB debug connector to use the rpiboot mode to boot the CM4 into USB mass-storage gadget mode which otherwise would give full read/write access to the eMMC of the CM4.
If this is is enough, just using the slight changes you started to make (and completing them) are sufficient and offer quite some protection. If this satisfies your requirements then stop reading here.
If local access to the CM4 USB-Gadget port can be disabled by our HW or physical access to the boards can be restricted to trusted persons, then that would be essentially equivalent.
Since the rest of the boot chain and update procedure would use the existing BalenaOS partitions and schemes, this does not protect against modifications done over the network (which could be hostile):
If due to some kind of security breach over the Network, an adversary gains privileged access to BalenaOS, a privileged container, a container which has access to your intellectual property, or a container with capabilities to access your intellectual property or the Balena host OS, these further changes would still be possible for an adversary (of course, these need another breach before they could be done, and require a really custom, targeted attack on your infrastructure, which is likely very expensive to do):
Remote modification of the BalenaOS Linux kernel
Remote modification of any other files on the boot, root and data partitions
Remote read of your container’s data if the remote access to the container or the Baleana host OS could be gained
Note: This 3rd point is not just a question of the security of BalenaOS and the Balena server and OS and application update infrastructure, but also a question of the whole chain of development and delivery of containers deployed on the CM4.
This page gives some steps to address the first three of these challenges:
Here is a copy of these steps, with comments by me:
Verified bootloader - Is taken care of by the CM4 secure boot example.
Kernel verification - Would be taken care of by the CM4 secure boot example, but:
Because BalenaOS uses U-Boot as another bootloader before starting the kernel, this does not verify the kernel, but the U-Boot binary, and also not it’s boot scripts.
Solution: Build boot.scr into U-Boot and make it verify a signed BalenaOS kernel image.
Root filesystem verification (dm-verity, IMA/EVM)
Not addressed by the CM4 Secure boot example.
It would have to be supported by BalenaOS.
dm-verity would be a good possibility.
IMA/EVM was made for enterprise server hardware and is very slow on embedded devices like the CM4.
Filesystem cryptography or block device encryption (dm-crypt)
The burnt OTP fuses inside the BCM2711 on the CM4 irrevocably limit the chip to only boot with your signed secure boot firmware. (the silicon die of it is your root of trust)
The signed firmware image loaded from the boot EEPROM only executes signed boot.img.
For BalenaOS, the signed boot.img contains the device tree overlay files and kernel8.img (in BalenaOS, it contains the Balena U-Boot which selects the root partition for booting).
Currently, Balena U-Boot loads the file boot.scr from the boot partition, which then boots BalenaOS. Because the contents in boot.scr control the boot process (selection which root partition to boot and loading the kernel from it), Balena U-Boot needs to be modified to include the contents of boot.src as an U-Boot macro which is compiled into U-Boot.
The boot.scr macros have to calculate a checksum over the BalenaOS kernel it loaded from the selected root partition and check that this checksum is signed before booting the kernel.
U-Boot has support for checking signatures using RSA, such step would have to be added.
Alternatively, U-Boot’s implementation of verified boot using signed FIT images could be used.
To verify the partitions before mounting them, the BalenaOS kernel needs use a (likewise verified or preferably built-in RAMdisk) to mount the root partition and it’s overlays.
To support this, the active/inactive sysroot partitions /dev/mmcblk0p2 and /dev/mmcblk0p3 would not be plain ext4 filesystems but dm-verity volumes, and the root hash/key for dm-verity would have to be provided in the verified initramfs (could be built into the kernel image)
Other partitions like the data partition mounted by BalenaOS would have to be verified as well for fully locking down the boot (likewise using dm-verity)
Depending on the application and if physical access is to storage is also a concern, data-at-rest on additional storage devices like a SATA/NVMe SSD (or network storage buckets) might have to be protected against unauthorized read/write by encrypting data at rest on these storage locations. This could be done by using encrypted files, an encrypted filesystem or dm-crypt. Since encryption layers do not strictly provide data integrity, a verification layer like dm-verity on the physical volume would complete the lock down.
Finally, if you need to trust that a remote device is in a trusted state (secure and verified boot was performed), one could remotely verify the checksums gathered during verified boot in a TPM and use TPM remote attestation for confidence that the remote TPM really recorded a verified boot and the checksums are not faked.
Of course how far you need to go (and in turn how massively you’d have to change BalenaOS into an OS capable of fully verified boot) depends on the threat you have at hand.
Security/Key architecture
Then comes the question on how to architect the deployment and updates.
If all devices share the same, single boot-loader signing key, the compromise of that key makes all devices vulnerable for changes and compromising your IP.
If each device uses it’s own unique bootloader signing key, each device also needs it’s own unique signed boot.img and boot.sig pair, and when delivering updates, these must of couse only get their matching updated pair of files.
Any failure on always delivering the correctly matching signed files/updates renders them not bootable, only to be recovered by booting a signed mass-storage recovery bootloader over USB as described here: usbboot/README.md at master · raspberrypi/usbboot · GitHub
Excursion into EFI land
Secure boot on the CM4 is really simple compared to EFI secure boot. For EFI secure boot, Microsoft established a very powerful but also very daunting, complicated model. EFI secure boot has 4 states of provisioning with transitions between them using a complicated state machine. It has key provisioning and the possibility to update keys, even thru the running OS. The OS makes calls to EFI running in system management mode behind the back of the OS. Using them and a big API, EFI stores variables for booting and the different layers of EFI secure boot keys.
For a full implementation, you need the shim boot-loader, the grub boot-loader and software infrastructure like MOK manager (machine owner key manager) when using MOK keys for development.
Further ideas
Based on the Secure boot Quickstart implementation, the BCM2711 appears to have only one set of fuses for one boot-loader key. Once the fuses are burnt, a bootloader key can no longer be changed or revoked. One should likely only burn the OTP fuses when the CM4 goes into production use.
When the OTP fuses are burnt, the bootloader key essentially becomes an “HW owners key”:
Who has the key, “owns” the inital step of loading and updating the bootloader as well as flashing the eMMC with new software over USB.
Therefore, for the later stages of a secure boot, I guess it good to use another key which you can replaced, updated and/or revoke for the purpose of transferring the right to sign a BalenaOS kernel from one user to another and/or revoking that right when needed.
Non-Bootloader keys for kernel and rootfs deployment, with revocation
For this you could store the certificates/public keys of many pre-created keys inside the signed boot.img, and only use one or a few of them at a time and if the device changes e.g. from one set of users to another, revoke the older keys by incrementing a counter in the eMMC rpmb partition where the count value of the counter is the number of old keys already revoked. The rpmb partition is specifically designed for this kind of roll-back prevention: It can be read, but it can only be updated by using the secret rpmb key with which the rpmb partition is provisioned by you. U-Boot has code to do all operations supported by with the eMMC rpmb partition:
Step where Kyle’s secure boot attempt fails to boot (and where no signature is checked):
@kwaremburg Kyle, the analysis on the boot.scr is this (newlines inserted by me):
root@BalenaOS-CM4# grep boot.scr /mnt/boot/kernel8.img
boot_scripts=boot.scr.uimg boot.scr
...
scan_dev_for_scripts=for script in ${boot_scripts}; do
if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then
echo Found U-Boot script ${prefix}${script};
run boot_a_script; echo SCRIPT FAILED: continuing...;
fi;
done
This means that Balena U-Boot looks for ${prefix}boot.scr in the path ${devtype} ${devnum}:${distro_bootpart}. If it is not found, it is skipped and fallbacks are executed instead.
To run a verified boot.scr, you’d have to replace the macro scan_dev_for_boot inside of the Balena U-Boot to not look for boot.scr (via scan_dev_for_scripts) on mmc 0:1 (you see the message Scanning mmc 0:1… from the macro scan_dev_for_boot in your log), but instead just contain the contents of boot.scr as part of the Macro, so it would look like this:
run resin_set_kernel_root;fdt addr ${fdt_addr};fdt get value bootargs /chosen bootargs;env set bootargs "${bootargs} ${resin_kernel_root} rootwait";fdt rm /chosen bootargs;load ${resin_dev_type} ${resin_dev_index}:${resin_root_part} ${resin_tmp_loadaddr} /boot/Image.gz;unzip ${fileaddr} ${resin_kernel_load_addr};booti ${resin_kernel_load_addr} - ${fdt_addr}
Yes, you have to build the BalenaOS U-Boot image file for this, so you need to run your your own BalenaOS image build starting from cloning GitHub - balena-os/balena-raspberrypi: Balena support for RaspberryPI boards and running balena-yocto-scripts/build/barys --machine raspberrypi4-64 on a strong server with a fast SSD with preferably around 40GB of space for the build.
Here is the definition of the scan_dev_for_boot macro which you have to replace:
It may only contain a copy of boot.scr, extendend with a signature check if you want of check the signature of the kernel (and so on…), and not run scan_dev_for_extlinux before, to not leave any door open where secure boot could be circumvented.
Thanks for the multiple replies and wealth of information you shared. I haven’t had a chance to fully digest and understand it all at a level where I would be able to implement it but I get the general ideas.
As far as protecting company IP we are well aware we need to secure the device while running (ssh, console, etc). This exercise is interested in preventing someone from pulling the CM4 or EEMC from the CM4 to gain access to the information on it.
dm-crypt and the secure boot functionality should get us there if I’m not mistaken.