Mounting SSD storage (Jetson tx2)

Product support
1

Hi,

I am trying to mount a SSD storage following the instruction in documents and the following topic,

I am using systemd and have enabled DBUS_SYSTEM_BUS_ADDRESS using the dashboard Device Configuration. The path for SSD is /dev/sda1. Here is my mount service,

[Unit]
Description = External SSD Drive

[Mount]
What = /dev/sda1
Where = /mnt/storage
Type = ext4
Options = rw,relatime,data=ordered

[Install]
WantedBy = multi-user.target

and in Dockerfile INISYSTEM is set to on,

#Enable container init system.
ENV INITSYSTEM on

#External storage
COPY mnt-storage.mount /etc/systemd/system/mnt-storage.mount
RUN systemctl enable /etc/systemd/system/mnt-storage.mount

I am not sure what I am missing since the SSD never gets mounted and the /mnt/storage path never gets created. The mount service is copied to the correct location,

:/etc/systemd/system# ll
total 28
drwxr-xr-x 1 root root 4096 Jan 10 19:51 ./
drwxr-xr-x 1 root root 4096 Dec 18 05:50 ../
drwxr-xr-x 2 root root 4096 Dec 18 05:49 getty.target.wants/
-rw-rw-r-- 1 root root  174 Jan 10 19:47 mnt-storage.mount
drwxr-xr-x 1 root root 4096 Jan 10 19:51 multi-user.target.wants/
lrwxrwxrwx 1 root root   31 Jan  9 23:07 sshd.service -> /lib/systemd/system/ssh.service
drwxr-xr-x 2 root root 4096 Dec 18 05:49 sysinit.target.wants/
drwxr-xr-x 2 root root 4096 Dec 18 05:50 timers.target.wants/

also symlink is created,

Created symlink /etc/systemd/system/multi-user.target.wants/mnt-storage.mount, pointing to /etc/systemd/system/mnt-storage.mount.

running systemctl status command returns not found error,

systemctl status mnt-storage.mount
● mnt-storage.mount
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

also tried to start the service in container and again returned not found error,

systemctl start  mnt-storage.mount
Failed to start mnt-storage.mount: Unit mnt-storage.mount not found.

What is incorrect in my configuration?

Thanks!

4

Hi there,
What version of balenaOS and device type?
What is the output of systemd-analyze verify test-storage.mount.?

5

Thank you for your response.
Device type: Nvidia Jetson TX2 (BETA)
Host OS Version: balenaOS 2.29.0+rev1

Here are the outputs,

root@xxxx:/# systemd-analyze verify test-storage.mount.
Failed to prepare filename test-storage.mount.: Invalid argument
root@xxxx:/# systemd-analyze verify mnt-storage.mount.
Failed to prepare filename mnt-storage.mount.: Invalid argument
root@xxxx:/# systemd-analyze verify /etc/systemd/system/mnt-storage.mount
root@xxxx:/#
6

Hi Farzad,

I’m not 100% certain but I think the mountpoint has to exist. Can you try adding RUN mkdir -p /mnt/storage to your Dockerfile?

7

Hi wrboyce,

tried it with existing mountpoint and it didn’t wok :frowning:.

Apparently, the service is enabled,

root@xxxxx:/# systemctl list-unit-files
UNIT FILE                                  STATE
proc-sys-fs-binfmt_misc.automount          static
dev-hugepages.mount                        static
dev-mqueue.mount                           static
mnt-storage.mount                          enabled
proc-sys-fs-binfmt_misc.mount              static
sys-fs-fuse-connections.mount              static
....
8

Ah, damn, worth a try I guess!

I’m stabbing in the dark a bit here, can you show me the output of systemctl status mnt-storage.mount and journalctl -ab -u mnt-storage.mount please?

What happens if you manually start the service (systemctl start mnt-storage.mount)?

9

here are the outputs,

root@xxxx:/# systemctl start mnt-storage.mount
Failed to start mnt-storage.mount: Unit mnt-storage.mount not found.

root@xxxx:/# systemctl status mnt-storage.mount
● mnt-storage.mount
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

root@xxxx:/# journalctl -ab -u mnt-storage.mount
No journal files were found.
-- No entries --

thank you for your time!

10

Hmm… How about if you try and mount the drive manually?

mount -t ext4 -o rw,relatime,data=ordered /dev/sda1 /mnt/storage

11

That’s the interesting thing I forgot to mention. I can manually mount the storage and use the storage. So the mount command works fine.

12

Right, so its definitely systemd then (surprise surprise!). Bit of a long shot, but have you tried a systemctl daemon-reload? I don’t think it should be necessary, but at this point I’d say it is worth a try!

13

The service is still inactive.

I guess a solution would be to add mounting command into startup script but I am more interested to know what is incorrect in my configuration.

14

I’m going to dig out a usb drive and try this with a device I’ve got here, it is puzzling me now. I’ll let you know how I get on.

19

Farzad, can you just confirm for me which base image you’re using so I know we’re both singing from the same hymn sheet?

Cheers.

25

arm64v8/ros:kinetic-ros-base-xenial is our base image

I can share the dockerfile with you if you need it.

thanks you!

26

Hi @wrboyce ,

Did you get a chance to look into this issue?

thank you !

27

Hi, since you are not using the Balena base images, a lot of things work differently, including systemd. We are still troubleshooting on your base image, as on our own ubuntu images work relatively straightfoward but with some required changes.

We are still looking into it, and will get back to you soon!

28

thanks!

29

Hi @Farzad, this turned out to be a lot deeper rabbit hole than we’ve expected.

TL;DR: you won’t be able to use systemd's .mount units to mount devices inside the container, see this reply to a similar question by systemd's creator: https://github.com/systemd/systemd/issues/6569#issuecomment-321172061

Will have to find a different way, for example, the start script you are using manually mounting the SSD for you as a step before you start your application. This would be our suggestion to do.

Nonetheless there are a few more comments that we’ve uncovered in the debugging section:

Base image

You are using an external base image, you mentioned arm64v8/ros:kinetic-ros-base-xenial. That base image does not know about the INITSYSTEM=on behaviour, which is a property of our resin/... base images. Just to clear that up, it wouldn’t have made a difference there.

System DBUS

You mentioned that the service ended up with this status:

systemctl status mnt-storage.mount
● mnt-storage.mount
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

It is because you modified the DBUS_SYSTEM_BUS_ADDRESS address for the entire application, and thus your systemd was talking to the host OS’s systemd, and seen the host’s service files/mount files, etc, and was not looking inside the container’s own services. For local services if you use systemd you cannot use the host’s DBUS, or end up checking the wrong systemd instance, in practice.

Systemd installation

Since you are using an external base image, if you do want to use systemd for other things, besides mount, you will have to modify how things are installed. You can see some guidance on this in the docs: https://www.balena.io/docs/reference/base-images/base-images/#installing-your-own-initsystem. One thing to note, that in my experience during this debugging with that ROS base image, I needed to comment out the VOLUME ["/sys/fs/cgroup"] line for things to work properly (adding a # in the front of that line).

These steps in that doc have a bunch of changes to make systemd work properly in a container environment.

udev

We also make udev work inside containers better (when udev is chosen to be run), including showing devices dynamically properly. This needs a few more changes, which are visible inside our base images as well, something along the line as the entry.sh script in this example https://github.com/balena-io-library/base-images/blob/3b667b1c97069ae9bca5ead7db5e040a87f62fd8/balena-base-images/armv7hf/ubuntu/xenial/run/entry.sh For full systemd+udev functionality this needs to be combined in a single entry.sh.

We have put together a repository that showcases systemd + udev used together with an external base image. It’s very much work in progress, but should give some pointers and we are working on better documenting it in the future.

(It’s very WIP, and this above is based on the base image you used, so expect that we make this a bit more generic in the future. We’v found the TL;DR resolution using this example above.)

Hope this helps, and thanks a lot for bringing this up! Let us know if you have any questions!