Hello!
We are currently using balenaOS on a Jetson Xavier NX for a couple of robots in the field. Everything works perfectly well with the Developer Kit including SD card.
However, with the chip shortage going on there seems to be no stock left of Xavier NX Developer Kits - which means we have had to switch to the eMMC production module and a third-party carrier board. Here’s where the issue comes in: the eMMC only has a capacity of 16 GB, but our running image - which includes ROS, CUDA, etc - is around 9GB. This means we can’t really do any updates on the device without running out of space.
This is why we opted to get an NVMe SSD that we attach to our carrier board’s M.2 slot, so we’d have enough room to store our data. The end goal here is to have the cached images, in the /var/lib/docker/overlay2 folder, sit on the SSD - thus leaving only the active container(s) on the eMMC storage.
Now I’ve seen that it’s impossible to boot out of anything other than the eMMC, so the direct option seems out of the window. After that, here’s what I tried so far:
Mounting the SSD from the Host OS
By remounting the filesystem I was able to mount the SSD in the Host OS:
mount -o remount,rw /
parted /dev/nvme0n1 mklabel gpt
parted -a opt /dev/nvme0n1 mkpart primary ext4 0% 100%
mkfs.ext4 -L nvme /dev/nvme0n1
mkdir -p /mnt/nvme && mount -o defaults /dev/nvme0n1 /mnt/nvme
To make the changes persist through reboots, I then tried adding the following line to fstab:
/dev/nvme0n1 /mnt/nvme ext4 defaults,sync 0 0
Doing a live reload of fstab using mount -a makes the drive show up, but after a reboot the entire device does not show up on balenaCloud. Am I doing something wrong here?
Changing the resinOS-flash file
Following the approach in Method for Mounting External Storage to Multiple Containers does not seem to work either - as soon as I make changes to the resinOS-flash.xml file (step 3 in the post) and use jetson-flash to flash the eMMC, the device again does not come online in balenaCloud.
The author said that it looks like a ‘more official solution is forthcoming’, and has suggested perhaps using a volume named resin-data-override to automatically add an external drive as a partition for the entire device. Is there any news on this?
Mounting the SSD in a container
Looking at Adding An NVMe Drive and Postgres Database Persistent Storage on NVMe SSD instead of SD Card, I managed to get the SSD mounted automatically in a bare-bones default container, using the script and Dockerfile posted there (thanks, @ts-cfield !):
Initialization shell script
#!/usr/bin/env bash
su - -c "mkdir -p /mnt/nvme" root
device=$(blkid | grep "LABEL=\"nvme\"" | cut -d : -f 1)
echo "Mounting device = ${device}"
su - -c "mount -t ext4 -o rw ${device} /mnt/nvme" root
# May want to call the parent image entrypoint instead
# exec docker-entrypoint.sh "$@"
exec "$@"
Dockerfile
FROM balenalib/jetson-xavier-nx-devkit-emmc
ENV UDEV=1
# Include the script in the image
COPY custom-init.sh /custom-init.sh
# Needed to make the script executable
RUN chmod +x /custom-init.sh
# The name of the initialization shell script can be anything
ENTRYPOINT ["/custom-init.sh"]
# Change to the command used by the parent image
CMD ["sleep", "infinity"]
Then, mounting the same volume in the Host OS gives me access to files put on there in the other running service. Nice. Next, I try to make a symlink between the stored images’ folder and the newly mounted drive:
root@ded122f:/var/lib/docker# ln -s /nvme overlay2
This works - /var/lib/docker/overlay2 now also gives me access to the test files I put into the SSD while ssh’d into the running container. After a powercycle, the symlink remains, but the drive itself is not mounted in the Host OS anymore.
I’m kind of at a loss of what to do next: I’d like to get the Host OS’ /var/lib/docker/overlay2 folder to somehow be symlinked to the SSD, but that seems impossible without persistent configuration of the drive on the Host OS - which seems impossible as well.
The author seems to agree, though their use case is slightly different:
“I had another idea to move the /var/lib/docker/volumes folder to the NVMe and then all Named Volumes would be stored on the larger, more stable (non-SD card) “external” drive. Moving the /var/lib/docker/volumes folder appears to be very involved and possibly harmful to the OS. Using symbolic links can also cause errors. While this would make persisting storage on an external drive relatively straight-forward, it is basically a non-starter.”
So, my question is: is there a way to automatically and persistently get the NVMe drive mounted on the Host OS, such that I can move the cached images from the Host OS to this drive automatically?
Thank you,
Peter
