Read/Write Root

Currently investigating implementation of a fully secure HostOS (SOC -> SPL -> boot loader -> BalenaOS -> Application). Trying to get the prototype running we ran into a problem.

Why does the root filesystem get remounted RW here?

Hi @SplitIce ,

That is a good question. I’m afraid I don’t have an obvious answer.

Have you tried to remove the remount to see what breaks? If nothing breaks, it could be leftover from original dev/debug work. Although, I suspect it could be a requirement before moving the mount and using pivot_root.

But I don’t see an obvious reason why it can’t be remounted as read-only just before the exec into systemd.

When the OS runs, this mount becomes /mnt/sysroot/active. And is indeed left as read-write.
While the OS container itself is mounted in / and is read-only.

Secure boot has come up in the past quite a few times.

Can you please elaborate a bit more about your approach and how you are going about prototyping it?

1 Like

Have you tried to remove the remount to see what breaks?

I’ll that test then. It was may plan anyway. I suspect it may be needed for $rootfs/counter but that’s it. We intend to move $rootfs/counter to resin-state or resin-boot anyway (as /active/counter and /inactive/counter). This will make the rootfs 100% read-only (unless I have missed any other rw files on /).

I was planning on offering to have a chat with you once we were done with the changes to discuss for your benefit. There are quite a few small architectural changes we have been forced to make to make a secure boot chain possible (at-least given our chosen setup / the restrictions of the ARM chip we use).

Can you please elaborate a bit more about your approach

As an overview:

Our hardware supports (al-be-it only documented with a single paragraph) a trusted boot mode with signature verification with efuses (and only one identified vulnerability with a subsequently verified workaround). This allows the hardware to verify the first 32K of boot code (SPL). From there we can verify the full loader (u-boot).

A modified u-boot verifies the kernel (and possibly the DTB if necessary in the future). The kernel verifies the root filesystem (signed code, signed hash) one kernel is available per rootfs. Kernel modules required for boot area all built-in (hence signed). The data partition is encrypted with a per device key (not stored on disk).

Supporting utilities such as the Host Update script & rollback scripts have had to be modified quite extensively as you can imagine. The process for both now looks a bit different. There is also hardening patches for some utilities (&configuration), the kernel and application specific.

We also need to investigate the resin-state partition, and the env file on the boot partition (code injection opportunity) to secure against possible trust chain breaks there.

We have had our own Yocto layer for device support for a couple years now (since we are on unsupported hardware) and a couple engineers with enough balenaOS & yocto experience to undertake this. This is prototyped as an extension of that repository for our hardware (although it would work on any Allwinner SoC I expect with minor alterations via a BSP and possibly even others).

Currently the two development projects, encryption and secure boot are meeting in the middle. The data encryption prototype is complete (only a few bugs remain that I’ll be investigating once secure boot is completed) and secure boot is up to mobynit. Once we finish the prototype we will move onto evaluation (performance, resource costs, architectural ramifications, etc) and testing (in particular we expect some board support bugs due to kernel upgrade - we require 5.1+ now).

Successfully booted in a readonly root today. Progress :slight_smile:

Ended up doing away with hostapp, it’s not really useful on a read-only root filesystem.

Successfully booted in a readonly root today. Progress :slight_smile:

Great!

I was planning on offering to have a chat with you once we were done with the changes to discuss for your benefit.

Yes it would be nice to have an in-depth conversation. Check pm for emails.

I’d also like to see the source code with your edits before the conversation if possible.

Regards,
ZubairLK

@zubairlk email sent

Realized I didn’t answer my original question.

The mount to read-write is necessary for two main reasons:

  1. The upper directory of overlayfs is required by specification to be read-writable. Despite being (re)mounted as read-only this check is performed in kernel (Invalid Argument errno).
  2. balena/docker load doesn’t create all the directory structure, and destruction of the machine deletes the merged directory after unmounting.

Due to this the sysroot is not really (and can’t be) readonly, even if the rootfs is mounted read-only. It may have been possible to mount upper via tmpfs and patch balena to create not cleanup the merged directory but in a situation like this where the sysroot needs to be strictly read-only (writing would invalidate the hash tree portion of the security model) it’s pointless.

We do hostupdates via direct write to the inactive sysroot anyway not via hostapp so we did away with it.

I see. Thanks for the update.

I look forward to seeing the thing working and its source code if possible