I figured out a method for mounting a large SSD as the storage partition for all docker containers. With this solution, any named volume (and even the docker image cache) are stored on the external SSD drive, instead of the normal partition on the boot drive.
I tested this on the Jetson TX2, but don’t see any fundamental reason why the same method wouldn’t work on other devices.
Background: BalenaOS uses disk labels to identify and mount the various partitions needed to run the OS. In particular, there is a partition called
resin-data that contains the docker image cache, and a directory for storing the data of any named (persistent) volumes created in your docker configuration.
The problem: BalenaOS doesn’t support mounting external storage to multiple containers. You can mount external storage inside of a single container at runtime, but sharing that mount between containers is not possible. This is a big bummer if you have a use case where one container is generating large amounts of data, and another container is responsible for uploading or processing that same data.
The solution: Create the
resin-data labeled partition on your external drive! Easy right? It’s actually not that hard, but it does require some customization of existing BalenaOS images and tools. It’s actually easier to accomplish this with ‘flasher’ images than it might be with regular images. Here are the steps I took for the TX2 (flasher image):
- Download an image from the dashboard (this method is verified to work with either prod or dev images).
- Clone the repo for the jetson-flash tool:
- In the jetson-flash tool, find the file at
assets/jetson-tx2-assets/resinOS-flash.xmland comment out the section of the XML that describes the
- Flash your image to the TX2 using the following jetson-flash command:
./bin/cmd.js -f <path/to/resin.img> -m jetson-tx2 -p -o <path/to/desired/output/dir>
(this flashes the TX2 but also keeps the intermediate files around on your host machine).
- Mount your external media to your host machine and determine which device it is (i.e.
/dev/sda). Use sgdisk to create a single large partition on the drive. (example script
partdisk.shat bottom of post).
- Rename the resin-data image on your host machine that was created in step 4:
cp /path/to/desired/output/dir/resin/resin-data /path/to/desired/output/dir/resin/resin-data.img(
ddin the next command doesn’t work right if the file doesn’t have the
- Copy the resin-data partition from step 6 to the partition you created in step 5 like so:
sudo dd if=path/to/desired/output/dir/resin/resin-data.img of=/dev/sda1
- Attach the external drive to your TX2 and boot!
If all goes according to plan, your device should now be using your external drive for all data storage.
- This will only work with drives that are connected to the system through an interface that can be accessed early on in boot. On the TX2 this includes SATA, but does not include NVMe. I think it will work with a USB connected drive, but have not verified this. The support for different interfaces likely varies from device to device.
- To accomplish this same task for a device that uses a regular (non-flasher) image, you will have to actually modify the image to pull out the
resin-datapartition. Perhaps someone with more familiarity can provide steps for that process in this thread.
I hope people find this helpful. Not being able to share a large data volume between containers was a showstopper for us. This solution allowed us to keep using Balena, which we really didn’t want to give up.
#!/bin/bash # partdisk.sh # Usage: ./partdisk.sh <device> # Example: ./partdisk.sh /dev/sda # Reference: https://www.rodsbooks.com/gdisk/sgdisk-walkthrough.html sgdisk -og $1 ENDSECTOR=`sgdisk -E $1` sgdisk -n 1:2048:$ENDSECTOR -c 1:"Resin Data" -t 1:8e00 $1 sgdisk -p $1