How to Mount an NVMe to Multiple Containers

Product support balenaOS
1

Thanks to a lot of help from Balena Support I have found a good solution for mounting an NVMe or an External Storage device to multiple containers. I have a container that generates a lot of data and files to an NVMe and I want other containers to access those files.

Currently BalenaOS doesn’t support mounting external storage to multiple containers. You can mount external storage inside of a single container at runtime, but sharing that mount between containers is not possible.

Solution: I followed a similar strategy to this repo from Balena: https://github.com/balena-io-experimental/balena-localvolume
The solution is to create a udev rule that auto mounts the device to the host. The config can be added manually to the config.json file as explained here: https://docs.balena.io/reference/OS/configuration/
For production devices I was able to use Balena’s Configizer tool: https://github.com/balena-os/configizer
This tool allows you to remotely update the config.json on any device you have access to. This was neccessary for backwards compatibility on production devices.

Here is an example of the udev rule that I used to auto-mount the NVMe to the host:

"os": {
    "udevRules": {
      "66": "ACTION==\"add\", SUBSYSTEMS==\"nvme\", KERNEL==\"nvme[0-9]n[0-9]\", ENV{ID_FS_USAGE}==\"filesystem\", RUN{program}+=\"/usr/bin/systemd-mount --no-block --automount=yes --bind-device --options=noexec,nosuid,nodev,sync --collect $devnode /run/mount/nvme\"\n"
    }
  }

Here is an example of the line I had to change in Balena’s Configizer tool to update the config remotely:

UDEVRULES[66]='ACTION==\"add\", SUBSYSTEMS==\"nvme\", KERNEL==\"nvme[0-9]n[0-9]\", ENV{ID_FS_USAGE}==\"filesystem\", RUN{program}+=\"/usr/bin/systemd-mount --no-block --automount=yes --bind-device --options=noexec,nosuid,nodev,sync --collect $devnode /run/mount/nvme\"\n'

Once the config is updated with the udev command I was able to create a new volume like this:

volumes:
  nvme-shared:
    driver: local
    driver_opts:
      type: none
      o: bind,rshared
      device: /run/mount

Then I was able to use that volume in any container and have access to all of my data on the NVMe!

2

Thanks for the update, I am glad the solution worked.