Hello,
Are the z and Z options supported for volumes in a docker-compose file?
Somehow my project containers seem to not be able to share data.
Thanks,
Marcelo
HI Marcelo, I don’t believe these options are supported at present.
Hi Marcelo, what is it you are trying to achieve? Sharing volumes between containers is possible if that will suit your needs?
Yes. I am trying to share volumes among all containers. The docker-compose I am using has those flags so I am wondering if those flags caused issues as the data seems not to be shared.
Following is a sample of a service in the docker compose file:
services:
security-secrets-setup:
image: edgexfoundry/docker-edgex-secrets-setup-go-arm64:1.2.0
container_name: edgex-secrets-setup
hostname: edgex-secrets-setup
command: “generate”
volumes:
- secrets-setup-cache:/etc/edgex/pki
- vault-init:/vault/init:z
- tmp-secrets:/tmp/edgex/secrets:z
- run-secrets:/run
I also have services that expose the volume to be shared but as read only as the example below:
vault:
image: vault:1.3.1
container_name: edgex-vault
hostname: edgex-vault
networks:
edgex-network:
aliases:
- edgex-vault
ports:
- “127.0.0.1:8200:8200”
cap_add:
- “IPC_LOCK”
entrypoint: ["/vault/init/start_vault.sh"]
environment:
- VAULT_ADDR=https://edgex-vault:8200
- VAULT_CONFIG_DIR=/vault/config
- VAULT_UI=true
volumes:
- vault-config:/vault/config
- vault-file:/vault/file:z
- vault-logs:/vault/logs:z
- vault-init:/vault/init:ro,z
- tmp-secrets-edgex-vault:/tmp/edgex/secrets/edgex-vault:ro,z
depends_on:
- consul
- security-secrets-setup
Note that the volumes are defined at the beginning of the docker-compose file.
Can I accomplish this in balenaOS?
Thanks,
Marcelo
Hey,
So we only support named volumes, not host-bound ones. I think the z flag is related to SELinux? which isn’t something we use.
If you were to do this:
volumes:
shared:
services:
first:
...
volumes:
- shared:/mnt/shared
second:
...
volumes:
- shared:/mnt/shared:ro
then the volume shared would be RO in the second service, but RW in the first. The data should be the same.
I have tried that but somehow the containers start but when I look at the logs, some of the containers are not able to access the data in the volumes.
I am attaching the docker-compose file I am using. Could you please let me know if there is something that is not supported by balenaOS? I really appreciate your help.
/*******************************************************************************
* Copyright 2020 Dell Inc.
* Copyright 2020 Intel Corporation.
*
* Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except
* in compliance with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software distributed under the License
* is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
* or implied. See the License for the specific language governing permissions and limitations under
* the License.
*
* @author: Leonard Goodell, Intel
* EdgeX Foundry, Geneva, version 1.2.0
* added: May 14, 2020
*******************************************************************************/
version: ‘2.1’
Note: Mongo has been deprecated in the Geneva (1.2.0) release.
Redis is the default database for Geneva (1.2.0).
Mongo will not be supported in future releases.
all common shared environment variables defined here:
x-common-env-variables: &common-variables
Registry_Host: edgex-core-consul
Clients_CoreData_Host: edgex-core-data
Clients_Notifications_Host: edgex-support-notifications
Clients_Metadata_Host: edgex-core-metadata
Clients_Command_Host: edgex-core-command
Clients_Scheduler_Host: edgex-support-scheduler
Clients_RulesEngine_Host: edgex-kuiper
Clients_VirtualDevice_Host: edgex-device-virtual
Databases_Primary_Type: mongodb
Databases_Primary_Host: edgex-mongo
Databases_Primary_Port: 27017
SecretStore_Host: edgex-vault
SecretStore_ServerName: edgex-vault
SecretStore_RootCaCertPath: /tmp/edgex/secrets/ca/ca.pem
Required in case old configuration from previous release used.
Change to “true” if re-enabling logging service for remote logging
Logging_EnableRemote: “false”
Clients_Logging_Host: edgex-support-logging # un-comment if re-enabling logging service for remote logging
volumes:
db-data:
log-data:
consul-config:
consul-data:
consul-scripts:
vault-init:
vault-config:
vault-file:
vault-logs:
non-shared volumes
secrets-setup-cache:
run-secrets:
tmp-secrets:
tmp-secrets-ca:
tmp-secrets-edgex-consul:
tmp-secrets-edgex-vault:
tmp-secrets-edgex-kong:
tmp-secrets-edgex-mongo:
tmp-secrets-edgex-security-proxy-setup:
tmp-secrets-edgex-support-notifications:
tmp-secrets-edgex-core-metadata:
tmp-secrets-edgex-core-data:
tmp-secrets-edgex-core-command:
run-vault:
services:
consul:
image: edgexfoundry/docker-edgex-consul-arm64:1.2.0
ports:
- “127.0.0.1:8400:8400”
- “127.0.0.1:8500:8500”
container_name: edgex-core-consul
hostname: edgex-core-consul
networks:
edgex-network:
aliases:
- edgex-core-consul
volumes:
- consul-config:/consul/config
- consul-data:/consul/data
- consul-scripts:/consul/scripts
- tmp-secrets-ca:/tmp/edgex/secrets/ca:ro
- tmp-secrets-edgex-consul:/tmp/edgex/secrets/edgex-consul:ro
- tmp-secrets-edgex-vault:/tmp/edgex/secrets/edgex-vault:ro
- tmp-secrets-edgex-kong:/tmp/edgex/secrets/edgex-kong:ro
environment:
- “SECRETSTORE_SETUP_DONE_FLAG=/tmp/edgex/secrets/edgex-consul/.secretstore-setup-done”
- EDGEX_DB=mongo
- EDGEX_SECURE=true
depends_on:
- security-secrets-setup
vault:
image: vault:1.3.1
container_name: edgex-vault
hostname: edgex-vault
networks:
edgex-network:
aliases:
- edgex-vault
ports:
- “127.0.0.1:8200:8200”
cap_add:
- “IPC_LOCK”
entrypoint: ["/vault/init/start_vault.sh"]
environment:
- VAULT_ADDR=https://edgex-vault:8200
- VAULT_CONFIG_DIR=/vault/config
- VAULT_UI=true
volumes:
- vault-config:/vault/config
- vault-file:/vault/file
- vault-logs:/vault/logs
- vault-init:/vault/init:ro
- tmp-secrets-edgex-vault:/tmp/edgex/secrets/edgex-vault:ro
depends_on:
- consul
- security-secrets-setup
security-secrets-setup:
image: edgexfoundry/docker-edgex-secrets-setup-go-arm64:1.2.0
container_name: edgex-secrets-setup
hostname: edgex-secrets-setup
command: “generate”
volumes:
- secrets-setup-cache:/etc/edgex/pki
- vault-init:/vault/init
- tmp-secrets:/tmp/edgex/secrets
- run-secrets:/run
vault-worker:
image: edgexfoundry/docker-edgex-security-secretstore-setup-go-arm64:1.2.0
container_name: edgex-vault-worker
hostname: edgex-vault-worker
environment:
- “SECRETSTORE_SETUP_DONE_FLAG=/tmp/edgex/secrets/edgex-consul/.secretstore-setup-done”
networks:
edgex-network:
aliases:
- edgex-vault-worker
volumes:
- vault-config:/vault/config
- consul-scripts:/consul/scripts:ro
- tmp-secrets:/tmp/edgex/secrets
- run-vault:/run
depends_on:
- security-secrets-setup
- consul
- vault
containers for reverse proxy
kong-db:
image: postgres:12.1-alpine
container_name: kong-db
hostname: kong-db
networks:
edgex-network:
aliases:
- kong-db
ports:
- “127.0.0.1:5432:5432”
environment:
- ‘POSTGRES_DB=kong’
- ‘POSTGRES_USER=kong’
- ‘POSTGRES_PASSWORD=${KONG_POSTGRES_PASSWORD:-kong}’
depends_on:
- security-secrets-setup
kong-migrations:
image: kong:2.0.4-ubuntu
container_name: kong-migrations
networks:
edgex-network:
aliases:
- kong-migrations
environment:
- ‘KONG_DATABASE=postgres’
- ‘KONG_PG_HOST=kong-db’
- ‘KONG_PG_PASSWORD=${KONG_POSTGRES_PASSWORD:-kong}’
command: >
/bin/sh -cx
‘until /consul/scripts/consul-svc-healthy.sh kong-db;
do sleep 1;
done && kong migrations bootstrap;
kong migrations list;
code=$$?;
if [ $$code -eq 5 ]; then
kong migrations up && kong migrations finish;
fi’
volumes:
- consul-scripts:/consul/scripts:ro
depends_on:
- consul
- kong-db
kong:
image: kong:2.0.4-ubuntu
container_name: kong
hostname: kong
networks:
edgex-network:
aliases:
- kong
ports:
- “8000:8000”
- “127.0.0.1:8001:8001”
- “8443:8443”
- “127.0.0.1:8444:8444”
tty: true
environment:
- ‘KONG_DATABASE=postgres’
- ‘KONG_PG_HOST=kong-db’
- ‘KONG_PG_PASSWORD=${KONG_POSTGRES_PASSWORD:-kong}’
- ‘KONG_PROXY_ACCESS_LOG=/dev/stdout’
- ‘KONG_ADMIN_ACCESS_LOG=/dev/stdout’
- ‘KONG_PROXY_ERROR_LOG=/dev/stderr’
- ‘KONG_ADMIN_ERROR_LOG=/dev/stderr’
- ‘KONG_ADMIN_LISTEN=0.0.0.0:8001, 0.0.0.0:8444 ssl’
restart: on-failure
command: >
/bin/sh -c
“until /consul/scripts/consul-svc-healthy.sh kong-migrations; do sleep 1; done;
/docker-entrypoint.sh kong docker-start”
volumes:
- consul-scripts:/consul/scripts:ro
depends_on:
- consul
- kong-db
- kong-migrations
edgex-proxy:
image: edgexfoundry/docker-edgex-security-proxy-setup-go-arm64:1.2.0
container_name: edgex-proxy
hostname: edgex-proxy
entrypoint: >
/bin/sh -c
“until /consul/scripts/consul-svc-healthy.sh kong; do sleep 1; done;
until /consul/scripts/consul-svc-healthy.sh security-secretstore-setup; do sleep 1; done;
/edgex/security-proxy-setup --init=true”
networks:
edgex-network:
aliases:
- edgex-proxy
environment:
<<: *common-variables
KongURL_Server: kong
SecretService_Server: edgex-vault
SecretService_TokenPath: /tmp/edgex/secrets/edgex-security-proxy-setup/secrets-token.json
SecretService_CACertPath: /tmp/edgex/secrets/ca/ca.pem
SecretService_SNIS: “edgex-kong”
volumes:
- consul-scripts:/consul/scripts:ro
- tmp-secrets-ca:/tmp/edgex/secrets/ca:ro
- tmp-secrets-edgex-security-proxy-setup:/tmp/edgex/secrets/edgex-security-proxy-setup:ro
depends_on:
- consul
- vault-worker
- kong
end of containers for reverse proxy
mongo:
image: edgexfoundry/docker-edgex-mongo-arm64:1.2.0
ports:
- “127.0.0.1:27017:27017”
container_name: edgex-mongo
hostname: edgex-mongo
entrypoint: >
/bin/sh -c
“until /consul/scripts/consul-svc-healthy.sh security-secretstore-setup; do sleep 1; done;
/edgex-mongo/bin/edgex-mongo-launch.sh”
networks:
- edgex-network
environment:
<<: *common-variables
volumes:
- db-data:/data/db
- consul-scripts:/consul/scripts:ro
- tmp-secrets-ca:/tmp/edgex/secrets/ca:ro
- tmp-secrets-edgex-mongo:/tmp/edgex/secrets/edgex-mongo:ro
depends_on:
- consul
- vault-worker
notifications:
image: edgexfoundry/docker-support-notifications-go-arm64:1.2.0
ports:
- “127.0.0.1:48060:48060”
container_name: edgex-support-notifications
hostname: edgex-support-notifications
networks:
- edgex-network
environment:
<<: *common-variables
Service_Host: edgex-support-notifications
SecretStore_TokenFile: /tmp/edgex/secrets/edgex-support-notifications/secrets-token.json
volumes:
- tmp-secrets-ca:/tmp/edgex/secrets/ca:ro
- tmp-secrets-edgex-support-notifications:/tmp/edgex/secrets/edgex-support-notifications:ro
depends_on:
- consul
- logging # uncomment if re-enabled remote logging
- mongo
- vault-worker
metadata:
image: edgexfoundry/docker-core-metadata-go-arm64:1.2.0
ports:
- “127.0.0.1:48081:48081”
container_name: edgex-core-metadata
hostname: edgex-core-metadata
networks:
- edgex-network
environment:
<<: *common-variables
Service_Host: edgex-core-metadata
Notifications_Sender: edgex-core-metadata
SecretStore_TokenFile: /tmp/edgex/secrets/edgex-core-metadata/secrets-token.json
volumes:
- tmp-secrets-ca:/tmp/edgex/secrets/ca:ro
- tmp-secrets-edgex-core-metadata:/tmp/edgex/secrets/edgex-core-metadata:ro
depends_on:
- consul
- logging # uncomment if re-enabled remote logging
- mongo
- notifications
- vault-worker
data:
image: edgexfoundry/docker-core-data-go-arm64:1.2.0
ports:
- “127.0.0.1:48080:48080”
- “127.0.0.1:5563:5563”
container_name: edgex-core-data
hostname: edgex-core-data
networks:
- edgex-network
environment:
<<: *common-variables
Service_Host: edgex-core-data
SecretStore_TokenFile: /tmp/edgex/secrets/edgex-core-data/secrets-token.json
volumes:
- tmp-secrets-ca:/tmp/edgex/secrets/ca:ro
- tmp-secrets-edgex-core-data:/tmp/edgex/secrets/edgex-core-data:ro
depends_on:
- consul
- logging # uncomment if re-enabled remote logging
- mongo
- metadata
- vault-worker
command:
image: edgexfoundry/docker-core-command-go-arm64:1.2.0
ports:
- “127.0.0.1:48082:48082”
container_name: edgex-core-command
hostname: edgex-core-command
networks:
- edgex-network
environment:
<<: *common-variables
Service_Host: edgex-core-command
SecretStore_TokenFile: /tmp/edgex/secrets/edgex-core-command/secrets-token.json
volumes:
- tmp-secrets-ca:/tmp/edgex/secrets/ca:ro
- tmp-secrets-edgex-core-command:/tmp/edgex/secrets/edgex-core-command:ro
depends_on:
- consul
- logging # uncomment if re-enabled remote logging
- mongo
- metadata
- vault-worker
networks:
edgex-network:
driver: “bridge”
Thanks,
Marcelo
some of the containers are not able to access the data in the volumes.
When they have started, use the web terminal to open a shell into the services and see what data is in the volumes. That will help determine the cause.
Are the volumes created at the host level or on each container? I was trying to look at the volumes but the paths are completely different. Please let me know if there are some special naming conventions.
Thanks
they are created in the /mnt/data/docker directory system on the host. There are no special convetions that I’m aware of.
The following is what is under the volumes folder. The names of the folders is not what is configured in the docker-compose as they are prefixed with some extra value.
Will balenaOS add that prefix when the container tries to access those volumes?
root@405d0d4:/mnt/data/docker/volumes# ls
1674084_consul-config
1674084_consul-data
1674084_consul-scripts
1674084_db-data
1674084_log-data
1674084_run-secrets
1674084_run-vault
1674084_secrets-setup-cache
1674084_tmp-secrets
1674084_tmp-secrets-ca
1674084_tmp-secrets-edgex-consul
1674084_tmp-secrets-edgex-core-command
1674084_tmp-secrets-edgex-core-data
1674084_tmp-secrets-edgex-core-metadata
1674084_tmp-secrets-edgex-kong
1674084_tmp-secrets-edgex-mongo
1674084_tmp-secrets-edgex-security-proxy-setup
1674084_tmp-secrets-edgex-support-notifications
1674084_tmp-secrets-edgex-vault
1674084_vault-config
1674084_vault-file
1674084_vault-init
1674084_vault-logs
361a98e3eade72a05418fd3f6790497fecdafda753905344902e06fc9e30d230
ba0b0a3c4a2a98fa00b2e248813a16a52aca72113de8e6e950be562a73e783d7
metadata.db
Thanks,
Marcelo
Hi there,
Just to clarify, that volume prefix is added by the supervisor and shouldn’t be interfering with what you are seeing. Are there specific files that you can observe existing in the volume in some containers but not in others? If you can provide some concrete examples, that would be very helpful to help you debug this.
Yes. Some volumes have data and some volumes are empty. I will need to debug one container at a time and see what is the source of the problem.
Thanks for your help.
Great, try to get a minimal working example and build up your application from there. Let us know if you can’t make it work we can look into it a bit more, but it should work just fine.