Best way to update files in shared named volume

Product support
1

Hello,

I have a multicontainer app with one service providing media data that is consumed by a media player through a shared named volume. My docker-compose.yaml is:

version: "2"
volumes:
  data-service:
services:
  multimedia-standalone:
    build: ./multimedia-standalone
    privileged: true
    group_add:
      - video
    volumes:
      - "data-service:/data-srv"
  data-service:
    build: ./data-service
    volumes:
        - "data-service:/data-srv"

The Dockerfile for the data-service is pretty simple:

FROM balenalib/raspberrypi4-64-alpine
COPY ./media /data-srv/media
CMD [ "balena-idle" ]

My desired functionality is to update media by rebuilding the image which does not happen as this is a shared volume. I’m wondering if anyone can suggest a best practice way to accomplish this?

2

Hey @richard.galvez

I think I see what you’re trying to do, so I’ll take a crack at the reasoning and a solution.

When you’re copying the ./media folder to /data-srv/media in your Dockerfile, that action is actually happening during the build phase of the image. That is the part where everything needed for the image is assembled and put together, before it reaches your device. So in this example, you’re trying to copy files to the final place on the device where your other container looks for them, but it’s happening before the media even reaches the device.

Depending on what you’re doing this may not be the best approach, but let’s get it working first! What you’d need to do is copy the media to a place within the image but outside the data directory, so for example COPY ./media /media. Then, when the container starts running, copy that media from /media to /data-srv/media. You could do this as a separate script and then change your CMD to run that script.

In that script you’d have something like:

#!/bin/bash
cp -R /media /data-srv
balena-idle

When organised like this it means the media will be in the image, and then when it runs it will be moved to the data partition on the device. You could also do it as part of your multimedia-standalone container, since there is no ongoing running process it could be combined into one.

As for if this is good practise or not will depend on what sort of media you’re distributing. It will cause the size of the image to increase, and would cause the same media to be distributed to every device in your fleet, and of course means that every time you want to add media you need to push and build a new release. Depending on the source of the media it may be better to host it somewhere else and have the device download it on container start - similar to how we used a script to copy it to the data partition, there could be a script that syncs the data partition with the source of your media.

I hope this helps but feel free to continue the discussion here either way!

1 Like
4

Interesting issue, had been looking at something similar myself recently. I went the same route as @chrisys in the end, but plenty of options worth thinking about (a Google of each will give the idea):

RSync
tar --compare
An NFS share
Renaming volume after each push (but you will end up with other volumes littering the device, would need to call a volume cleanup from time to time)
Use an online source
…

1 Like
5

Also:

  • Symlinks ← haven’t tried this one, would be very curious if it works. Could potentially be the best option. Wouldn’t benefit from the volume driver of the mount which can help with write processes if that is important, but unlikely to be an issue.
  • Storing the content in a .iso file and mounting that file in to the volume.
6

And here is an example RSync script I used for something else that could work as a template:

#!/usr/bin/env
destination=/app/
source=dist/

echo "Running in hot reload mode. Setting up the development environment..."
start_time=$(date +%s)

rsync --archive --delete --inplace $source $destination

echo "Development environment ready in $(($(date +%s)-$start_time)) seconds."

# Start Supervisor
exec supervisord -c supervisord.conf
7

Thanks all for your input! I ended up implementing what @chrisys suggested, interfacing with cloud storage to pull the data from.

Thanks!

1 Like