Multi container and shared volume data

Product support
1

Currently I got a fairly big monolithic balena image which is using systemd to run various services. This works as expected, but it would be nice to split it up into several smaller containers.

The split up would create the following:
One-to-many containers which would contain data only ( not a named volume and not a bind volume )
One-to-many containers which would contain self-contained services
One container for the main application mounting data from data only containers

The main issue from these requirements are the data only container. The balena documentation states that the balena backend only supports a subset of the docker-compose version 2.1. This subset does not support volumes_from docker-compose field which seems to be the field to use in order to share data between containers.

I have done some named volume experiments based on the multicontainer-getting-started.

The experiments include named volumes which are mounted as show in:

Multi containers using named volumes for data sharing

`version: ‘2’

volumes:
fundata: {}
proxydata: {}

services:

proxy:
build: ./haproxy
depends_on:
- frontend
- data
ports:
- “80:80”
volumes:
- proxydata:/proxy:rw

data:
build: ./data
expose:
- “8080”
volumes:
- fundata:/usr:rw

frontend:
build: ./frontend
expose:
- “80”
volumes:
- fundata:/tmp/fundata:rw
- proxydata:/tmp/proxydata:rw`

Question:

Is named volumes show in the above docker-compose how data should be shared between containers? Unsure if this method introduce data race condition depending on which contain starts first. Additionally it seems the any data pointed to the named volume will be copied and not mounted. May be an issue for larger data amounts.

Is there any plan to support volumes_from?

Is there a more optimal way to perform data shared between multi containers?

3

Hey @aliasbits

Yes, that’s currently the supported method right now.

You can use the docker-compose depends_on field to ensure the dependee starts first.

I’m not entirely sure what you mean by this, the volumes are mounted onto containers.

It’s not currently on the roadmap, but it is perhaps something that we can look into. We added something similar recently (network_mode: "service: <name>") which means that the required infrastructure is mostly there.

I would say what you have right now is the ideal method given the current setup of a balena device.

Let me know if you’d like further clarification on any of these points.

5

Hey CameronDiver

Clarification of: “Additionally it seems the any data pointed to the named volume will be copied and not mounted.”

If I boot up with a empty named volume, then I assume that anyone container which provide files for this volume must be copied from the container file system to the named volume in order for other containers to access this in the shared named volume.

example:
name volume A

container X
named volume mapping A:/opt

container Y
named volume mapping A:/opt

if container X provides files for /opt, then I would assume these must be copied to named volume before contain Y can see these, hence my “copy statement instead of mounting”

This statement may be wrong

6

It’s kind of like that yeah, so you would copy files into the named volume (or download directly into there etc), but because this is a filesystem mounted onto the container filesystem, these files will automatically be available from another container which also has the volume mounted. Does this make sense?

8

Yes.

Based on my current knowledge I would still assume docker compose volumes_from is a more optimal solution if a lot of data needs to be shared, but until then the above solution will be valid.

Thanks for feedback