# How to avoid adding config files to the docker images **URL:** \ **Category:** balenaOS\ **Created:** [September 13, 2019, 1:30pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065 "2019-09-13T13:30:42Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![krasi-georgiev](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@krasi-georgiev](https://forums.balena.io/u/krasi-georgiev)\ **Post date:** [September 13, 2019, 1:30pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/1 "2019-09-13T13:30:42Z")
Most docker images use config files mounted from the host which doesn’t work with the balena workflow and using a compose file. I am trying to come up with a way to create files from an env variable so that these can be used when starting a docker image. This will allow to use upstream images and not keep any config in the docker image itself making those more universal and provide new configs at runtime. For example: config\_env = “some config text” - env var set in the balena GUI start a docker container like `docker run prometheus/prometheus --config.file=$(echo $config_env > /env_files/config_env && echo /env_files/config_env` I am not sure if this would work , but will test it shortly. Do you think it would be possible to have this as part of balena os/cli? when you have an env var as `config_env` you can directly use it as a file `/env_files/config_env` ---
**Author:** ![hedss](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/hedss/32/210_2.png) [@hedss](https://forums.balena.io/u/hedss)\ **Post date:** [September 13, 2019, 2:20pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/2 "2019-09-13T14:20:54Z")
Hi @krasi-georgiev , Your solution should work without issues as the Supervisor will inject the environment variables you’ve defined into the service before it runs and another potential way is to include them encoded as part of a `docker-compose.yml` file. We don’t currently have any plans to allow environment variables to be mounted as part of the filing system, however. Another potential way to solve this is to retrieve configuration from a remote server on startup, which I believe some other customers do. Best regards, Heds ---
**Author:** ![krasi-georgiev](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@krasi-georgiev](https://forums.balena.io/u/krasi-georgiev)\ **Post date:** [September 13, 2019, 7:48pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/4 "2019-09-13T19:48:51Z")
> [@hedss](#): > > Your solution should work without issues as the Supervisor will inject the environment variables you’ve defined into the service before it runs and another potential way is to include them encoded as part of a `docker-compose.yml` file. We don’t currently have any plans to allow environment variables to be mounted as part of the filing system, however. thanks will try and report > [@hedss](#): > > Another potential way to solve this is to retrieve configuration from a remote server on startup, which I believe some other customers do. The device won’t be connected to the internet at all times so can’t use this solution. ---
**Author:** ![krasi-georgiev](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@krasi-georgiev](https://forums.balena.io/u/krasi-georgiev)\ **Post date:** [September 15, 2019, 10:38am UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/6 "2019-09-15T10:38:31Z")
unforunately this didn’t work: `ValidationError: Bind mounts are not allowed` here is the compose file entry: ```auto prometheus: image: prom/prometheus ports: - 9090:9090 volumes: - $(echo $PROMETHEUS_CONFIG > ./PROMETHEUS_CONFIG && echo./PROMETHEUS_CONFIG:/etc/prometheus/prometheus.yml) ``` could you suggest some other workaround? ---
**Author:** ![lucianbuzzo](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/lucianbuzzo/32/615_2.png) [@lucianbuzzo](https://forums.balena.io/u/lucianbuzzo)\ **Post date:** [September 16, 2019, 9:19am UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/7 "2019-09-16T09:19:55Z")
One workaround for this would be to use the target docker image as a base, and start the container with a script that first creates the config file from the `PROMETHEUS_CONFIG` env var before starting the service. For example something like this could work (untested): `Dockerfile` ```auto FROM prom/prometheus:latest WORKDIR /usr/src/app COPY run-prom.sh ./ CMD ["./run-prom.sh"] ``` `run-prom.sh` ```auto echo $config_env > /env_files/config_env ./bin/prometheus --config.file=/env_files/config_env ``` ---
**Author:** ![krasi-georgiev](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@krasi-georgiev](https://forums.balena.io/u/krasi-georgiev)\ **Post date:** [September 16, 2019, 10:16am UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/9 "2019-09-16T10:16:23Z")
This means maintaining a copy of lots of upstream images which is what I want to avoid at all cost. Maintaining these means updating versions, dealing with multi arch images etc. Can you think of any other workaround or would you consider adding this feature to the balena cli tool or maybe just removing the ValidationError would allow to use the workaround I suggested. ---
**Author:** ![lucianbuzzo](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/lucianbuzzo/32/615_2.png) [@lucianbuzzo](https://forums.balena.io/u/lucianbuzzo)\ **Post date:** [September 16, 2019, 10:39am UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/10 "2019-09-16T10:39:45Z")
I can’t think of any other workarounds in this instance, though this does seem to be the approach many people are taking to get around prometheus not accepting environment variables as configuration. ---
**Author:** ![krasi-georgiev](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@krasi-georgiev](https://forums.balena.io/u/krasi-georgiev)\ **Post date:** [September 16, 2019, 11:10am UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/12 "2019-09-16T11:10:40Z")
It is not just Prometheus I am using few other images in this project and none of them allow using env variables as configurations. I think all upstream images that I have used so far don’t use env variables as configs(just for some simple cli args like username, pass etc). In the balena worlds I think this is a great limitation and doesn’t allow reusing the same image for a different project as the config needs to be baked in the image. ---
**Author:** ![robertgzr](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/robertgzr/32/1387_2.png) [@robertgzr](https://forums.balena.io/u/robertgzr)\ **Post date:** [September 16, 2019, 5:03pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/13 "2019-09-16T17:03:19Z")
As an alternative, but very similar, solution to that you can run little sidecar container that sets up those environment variables on a volume mount for you and then share them with the appropriate services. Make sure you make the prometheus service dependent on the config- sidecar. ---
**Author:** ![krasi-georgiev](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@krasi-georgiev](https://forums.balena.io/u/krasi-georgiev)\ **Post date:** [September 16, 2019, 6:52pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/15 "2019-09-16T18:52:04Z")
Thanks that is a good suggestion, I will try it. This will allow to use upstream images without any changes. Btw what is the reason you don’t want to add this feature to balena os or the balena cli? ---
**Author:** ![hedss](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/hedss/32/210_2.png) [@hedss](https://forums.balena.io/u/hedss)\ **Post date:** [September 16, 2019, 7:02pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/16 "2019-09-16T19:02:35Z")
Hi @krasi-georgiev , One thing to note with the sidecar solution is you’ll need to ensure the applications using the volumes configured by the sidecar keep testing for the file until it’s available, as a dependency on another service container only ensures that the container has started, and not that a process has for example initiated. Most of our customers use either hardcoded values in their docker-compose files or set via the dashboard and CLI for use cases like this. However, you’re right in that this isn’t a great case for when there’s no guarantee that there will be an internet connection. In general, we don’t like adding any features to our docker-compose schema that don’t map to standard docker-compose features. However, I do wonder in this case if there’s another we could do it for offline support. I’m going to raise an issue internally to discuss this use case. We’ll let you know when we have more information on this discussion. Best regards, Heds ---
**Author:** ![krasi-georgiev](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@krasi-georgiev](https://forums.balena.io/u/krasi-georgiev)\ **Post date:** [September 16, 2019, 7:26pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/19 "2019-09-16T19:26:39Z")
> [@hedss](#): > > One thing to note with the sidecar solution is you’ll need to ensure the applications using the volumes configured by the sidecar keep testing for the file until it’s available, as a dependency on another service container only ensures that the container has started, and not that a process has for example initiated. Hm , yeah I will try different ways with the sidecar and will report if anything worked. > Most of our customers use either hardcoded values in their docker-compose files or set via the dashboard and CLI for use cases like this. I didn’t quite understand what you mean by this? You can only set env variables there and not files. > In general, we don’t like adding any features to our docker-compose schema that don’t map to standard docker-compose features. I don’t think the official docker-compose tool disallows host bind mounts so maybe just removing this restriction in the balena cli will allow using my original suggestion. > I’m going to raise an issue internally to discuss this use case. We’ll let you know when we have more information on this discussion. thanks, greatly appreciated! 👍 ---
**Author:** ![hedss](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/hedss/32/210_2.png) [@hedss](https://forums.balena.io/u/hedss)\ **Post date:** [September 16, 2019, 7:39pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/20 "2019-09-16T19:39:14Z")
Hi again, > > Most of our customers use either hardcoded values in their docker-compose files or set via the dashboard and CLI for use cases like this. > > I didn’t quite understand what you mean by this? You can only set env variables there and not files. So you could, for example, base64 a config file in an environment variable, and on service startup decode this variable and then pipe it to a named volume. Whilst you’re absolute right you can use host bind mounts, we don’t support this functionality because it can potentially be extremely easy for people to bind something vital into the container and then put the system into a state that we couldn’t recover. Hopefully we can come to a good solution for you when the issue’s discussed! Best regards, Heds ---
**Author:** ![krasi-georgiev](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@krasi-georgiev](https://forums.balena.io/u/krasi-georgiev)\ **Post date:** [September 16, 2019, 7:42pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/22 "2019-09-16T19:42:20Z")
> [@hedss](#): > > So you could, for example, base64 a config file in an environment variable, and on service startup decode this variable and then pipe it to a named volume. I think this would work for me as well, could you give an example within a compose file? ---
**Author:** ![imrehg](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/imrehg/32/36_2.png) [@imrehg](https://forums.balena.io/u/imrehg)\ **Post date:** [September 16, 2019, 8:10pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/27 "2019-09-16T20:10:26Z")
The flow would be something like this. Imagine, that your config file has this contents: ```auto value1=one value2=two value3=three ``` (can be anything for this example, and added spaces, just to show that it works too). Now, on your work machine, if you use Linux, for example, you can use the `base64` tool to encode this file. Say the file is called `some.cfg`, you would go: ```auto ~> base64 some.cfg dmFsdWUxPW9uZQoKdmFsdWUyPXR3bwoKdmFsdWUzPXRocmVlCg== ``` meaning that string above is the base64-encoded version of the file. Add this as an environment variable, let’s call it `MY_SOME_CFG`, in your `docker-compose.yml` you could add: ```auto ... servicename: ... environment: - MY_SOME_CFG="dmFsdWUxPW9uZQoKdmFsdWUyPXR3bwoKdmFsdWUzPXRocmVlCg==" ... ``` where the `...` is just other fields. Then in your start script, you can do: ```auto echo "$MY_SOME_CFG" | base64 -d > /path/to/some.cfg ``` which would decode your environment variable, and save it to that path set above. How does this sound? The advantage of this as well, that by setting alternate values in the dashboard to `MY_SOME_CFG` you can override the ones set in the `docker-compose.yml` (the compose value is overridden by the fleet variable, which is overriden by the device variable). You can try out encoding using other tools, such as this website, if that’s more practical: [https://www.base64encode.org/](https://www.base64encode.org/) (not endorsing the site, just thinking that it might be helpful if you don’t have a Linux development machine, and want to see how [base64](https://en.wikipedia.org/wiki/Base64) encoding/decoding works) ---
**Author:** ![krasi-georgiev](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@krasi-georgiev](https://forums.balena.io/u/krasi-georgiev)\ **Post date:** [September 16, 2019, 8:16pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/29 "2019-09-16T20:16:08Z")
This is more or less what I already tried and the problem is how would you mount `/path/to/some.cfg` inside a container (lets say prometheus for this example)? You need to write this file on the host, but mounting files from the host is disabled. ---
**Author:** ![hedss](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/hedss/32/210_2.png) [@hedss](https://forums.balena.io/u/hedss)\ **Post date:** [September 16, 2019, 8:32pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/31 "2019-09-16T20:32:15Z")
Many apologies, but I think maybe we’ve not explained the volume storage correctly. Whilst there is indeed no bind mounting of volumes for the host, you can define named storage volumes that are persistent and can be shared between services. What we’re suggesting is: 1. Store configuration as encoded environment variable ‘somewhere’ (docker-compose file, UI, etc) 2. Decode configuration file and store into a persistent named storage volume on the host. You can bind any path in the container to the named volume, for example in your docker-compose file: ```auto volumes: storage-volume: services: prometheus: image: ... volumes: - 'storage-volume:/path/to' ``` 1. `echo "$MY_SOME_CFG" | base64 -d > /path/to/some.cfg` in your entry script for your service. This now contains a configuration that is persistent across reboots, etc., is bound to the `/path/to` directory within the prometheus service and can be used by any other service by using the `storage-volume` persistent volume. We’re not sure why you need to store any data on the host FS, as there shouldn’t be a requirement to do so for any services. Does this make sense? Could you please explain a bit about your thinking for using host FS space? Best regards, Heds ---
**Author:** ![krasi-georgiev](https://avatars.discourse-cdn.com/v4/letter/k/ccd318/32.png) [@krasi-georgiev](https://forums.balena.io/u/krasi-georgiev)\ **Post date:** [September 16, 2019, 8:43pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/33 "2019-09-16T20:43:23Z")
> [@hedss](#): > > `echo "$MY_SOME_CFG" | base64 -d > /path/to/some.cfg` in your entry script for your service. This now contains a configuration that is persistent across reboots, etc., is bound to the `/path/to` directory within the prometheus service and can be used by any other service by using the `storage-volume` persistent volume. Yes I am aware of the storage volumes and I am using these to persists data, but I am missing some points on how would you write this config file and use it without using some custom Prometheus image with a custom entrypoint? Using a sidecar service that converts all env variables to files and writing them to a shared volume was a good idea, but there is no guarantee that prometheus will not start before these files are actually written. Anyway I will try this suggestion and will report back if I could make it work as expected. ---
**Author:** ![hedss](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/hedss/32/210_2.png) [@hedss](https://forums.balena.io/u/hedss)\ **Post date:** [September 16, 2019, 9:02pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/34 "2019-09-16T21:02:38Z")
Hi again, Sorry, I understand now. Yes, you’d need to create a new Docker image based on the Prometheus image that used the custom script. Best regards, Heds ---
**Author:** ![imrehg](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/imrehg/32/36_2.png) [@imrehg](https://forums.balena.io/u/imrehg)\ **Post date:** [September 16, 2019, 9:04pm UTC](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065/35 "2019-09-16T21:04:07Z")
Yeah, that is a tricky thing, regarding the timing of the updates. The previous suggesion was using a pattern where there’s a start script that runs in a service and does some logic, then starts up your main process. I guess here you are using an upstream/official Prometheus container? Maybe you can combine this sidecar (and shared volume) patter with triggering a prometheus config reload through its API? I’ve found this blogpost [https://www.robustperception.io/reloading-prometheus-configuration](https://www.robustperception.io/reloading-prometheus-configuration) so could I guess call `/-/reload` endpoint on the prometheus container? If the networking set up properly and container networking is used, the service name can be used so say `prometheus` service could be triggered with `curl -X POST http://prometheus:9090/-/reload` (ports might need to be allowed as well, but worth a try experimenting with it). Something along these lines. Does this make sense? [Next page](https://forums.balena.io/t/how-to-avoid-adding-config-files-to-the-docker-images/32065.md?page=2)