Method for Mounting External Storage to Multiple Containers

Product support balenaOS
1

I figured out a method for mounting a large SSD as the storage partition for all docker containers. With this solution, any named volume (and even the docker image cache) are stored on the external SSD drive, instead of the normal partition on the boot drive.

I tested this on the Jetson TX2, but don’t see any fundamental reason why the same method wouldn’t work on other devices.

Background: BalenaOS uses disk labels to identify and mount the various partitions needed to run the OS. In particular, there is a partition called resin-data that contains the docker image cache, and a directory for storing the data of any named (persistent) volumes created in your docker configuration.

The problem: BalenaOS doesn’t support mounting external storage to multiple containers. You can mount external storage inside of a single container at runtime, but sharing that mount between containers is not possible. This is a big bummer if you have a use case where one container is generating large amounts of data, and another container is responsible for uploading or processing that same data.

The solution: Create the resin-data labeled partition on your external drive! Easy right? It’s actually not that hard, but it does require some customization of existing BalenaOS images and tools. It’s actually easier to accomplish this with ‘flasher’ images than it might be with regular images. Here are the steps I took for the TX2 (flasher image):

  1. Download an image from the dashboard (this method is verified to work with either prod or dev images).
  2. Clone the repo for the jetson-flash tool: https://github.com/balena-os/jetson-flash
  3. In the jetson-flash tool, find the file at assets/jetson-tx2-assets/resinOS-flash.xml and comment out the section of the XML that describes the resin-data image.
  4. Flash your image to the TX2 using the following jetson-flash command:
    ./bin/cmd.js -f <path/to/resin.img> -m jetson-tx2 -p -o <path/to/desired/output/dir>
    (this flashes the TX2 but also keeps the intermediate files around on your host machine).
  5. Mount your external media to your host machine and determine which device it is (i.e. /dev/sda). Use sgdisk to create a single large partition on the drive. (example script partdisk.sh at bottom of post).
  6. Rename the resin-data image on your host machine that was created in step 4:
    cp /path/to/desired/output/dir/resin/resin-data /path/to/desired/output/dir/resin/resin-data.img (dd in the next command doesn’t work right if the file doesn’t have the .img extension).
  7. Copy the resin-data partition from step 6 to the partition you created in step 5 like so:
    sudo dd if=path/to/desired/output/dir/resin/resin-data.img of=/dev/sda1
  8. Attach the external drive to your TX2 and boot!

If all goes according to plan, your device should now be using your external drive for all data storage.

Caveats:

  • This will only work with drives that are connected to the system through an interface that can be accessed early on in boot. On the TX2 this includes SATA, but does not include NVMe. I think it will work with a USB connected drive, but have not verified this. The support for different interfaces likely varies from device to device.
  • To accomplish this same task for a device that uses a regular (non-flasher) image, you will have to actually modify the image to pull out the resin-data partition. Perhaps someone with more familiarity can provide steps for that process in this thread.

I hope people find this helpful. Not being able to share a large data volume between containers was a showstopper for us. This solution allowed us to keep using Balena, which we really didn’t want to give up.

#!/bin/bash

# partdisk.sh
# Usage: ./partdisk.sh <device>
# Example: ./partdisk.sh /dev/sda

# Reference: https://www.rodsbooks.com/gdisk/sgdisk-walkthrough.html

sgdisk -og $1
ENDSECTOR=`sgdisk -E $1`
sgdisk -n 1:2048:$ENDSECTOR -c 1:"Resin Data" -t 1:8e00 $1
sgdisk -p $1
1 Like
4

Hi, not quite sure if what you are doing is future-safe as I have heard this method of using labels to identify balena partitions being discussed. We might move to uuids or partuuids in the future as labels can cause confusion and failure to boot caused by exactly the kind of manipulation you are playing with.

I am currently doing a similar thing mounting a USB drive partition from within a (privileged) container to have persistent storage on an external media but that only really works for that container as other containers can not see the mount or rely on the partition to be mounted. Basically that way every container needs to mount its own partition which amounts to a terrible waste of disk space.

7

@samothx totally understood. In my conversations with Balena support it sounds like a more official solution is forthcoming. I’d love to know how progress is coming on an official way to mount external storage to multiple containers. This solution took me a while to figure out so I figured I’d leave it here in case someone else was in the same situation I was.

Certainly mounting an external partition within one container is a fine option if it works for your use case. But it doesn’t work for use cases where one container is consuming data produced by another (large video files in my case).

1 Like
8

A thought: perhaps an official solution could use disk labels in a different way? It would require modifying BalenaOS, but perhaps if one were to label a disk as resin-data-override it could supersede the default resin-data partition as the storage partition used for named volumes in the docker config. Could be a relatively elegant and simple solution. It would also allow the use of NVMe drives (and pretty much anything else the OS can mount).

10

Thanks for this, I’m also doing a similar solution to share an SSD on the TX2 with multiple containers. If there’s official support on the roadmap I’d certainly be interested. I’ll watch this thread for now for updates.

12

Thank you for these inputs, we will take them into account.