Yeah and maintaining a copy of an upstream image is one thing I want to avoid - versioning, different architectures etc.
I am using few upstream images in our project and will try all other suggestions before sourcing to creating a custom one with an entrypoints.sh script.
Yes I am aware of this and will probably work, but I was hoping for a more generic approach which I can use for all upstream images used in the project. Prometheus was just for the example, but I use another 2-3 images which are facing the same problem.
btw Thanks for the dedicated support and the different suggestions.
I think this is a pretty generic approach, that can be implemented in images. Triggering configuration reload is more and more common thing in current software, exactly for similar use cases - updating configuration without container restart.
What other services are you using, for reference?
It’s a tricky problem in general since there are so many constraints (ie. if there could be a start script, things are solved; if the config would be a single env var, there’s no need for this convolution)… So interesting problem, hopefully this above will give some hints to how it should be solved in this particular setting.
I am also using AlertManager and LoraServer(which consist of few images that load config files.)
I would think that a more generic approach would look something like:
Deploy a side car that monitors all env variables with a suffix _file eg. env_name_file and creates a file from the content in a shared volume.
Than all services can mount this shared volume.
At the moment all services are restarted when an env variable changes so essentiallt don’t even need to send a reload request as they would be restarted anyway.
The only problem to solve is how to ensure that all services start only after the sidecar is fully started and has written all env vars to files…
In my experience, the services I’ve seen are going in the direction of explicit reload requests, so I think the industry considers that the more generic solution. What you describe is indeed good, and there are cases when that sort of file-change monitoring is applicable as well. To implement that on your side would likely more change on the existing projects. If you are using any project that you can modify, you could use e.g. inotify based tooling for that, for example, and implement that you described. But that really depends on the specifics of your project. There’s more than one way to solve things, and unfortunately, the same thing is not applicable (or accepted) by all upstream projects.
This is a very interesting question, though, so thanks for raising it! And while I don’t think we can solve it in balena in a generic way, there might be things we can improve on, with time and experience!
Not really, as e.g. in the kubernetes domain it’s similar, environment variables are part of configuration, and then the services are redeployed with the new config. for example.
And if the services you use can ingest single line env vars, this “restart” model would work too.
In the balena space the the containers need to be restarted on change so that the container can have the new env vars (as that doesn’t happen dynamically) and internal code running can pick it up.
I understand that it’s all quite confusing, I think the entire industry is creating new mental models to deal with these use cases, so there’s a lot more to come
Yeah, we understood that. Hence our suggestion to do the sidecars + shared volumes + reload signals. But you outlined another viable (pretty generic) pattern and gave suggestion that when you are not using upstream, but make your own project, it would be a good place for that. Haven’t suggested doing that for cases when you use upstream containers unmodified.
Having said that, in practice we rarely see that all services in a solution are upstream. Your setup of prometheus + alertmanager + LoRa can very well be one of them, and in which case have to work with what you have (and sidecar + … approach is possibly viable).
Thus suggest following up as you mentioned before, actually implementing one of these in practice and seeing if it solves your use case.
Yes this is what I meant exactly. WIth the restarts on every env change don’t need to send config reload signals and not all binaries have a http endpoint to trigger a reload so I don’t think it is the right direction. I will test with the last suggestion with the side car that creates the files from env vars and writes these in a shared volume.
I think that all images by default exit if a specified config file doesn’t exist so this will ensure that even if they start before the config file is created they will keep restarting until the file is there.
That’s just the point. We won’t modify upstream Prometheus images to work with “configuration from environment variables”. One has to either adapt to the project, or adapt the project. From the above discussion your path forward is adapting to the project.
That might or might not be an intended behaviour sometimes (container restart loops are rarely good in my experience). And this would happen only on first start - since you put the file into the shared volume, after the first time the file is there, there is a configuration. But there’s still a race, if you change that env var, to update the config file, unless you send the reload signal, it might not be picked up automatically.
But don’t want to discuss this to oblivion. There are patterns that are good, there are patterns that are not that good, and many common projects use different patterns for common tasks such as configuration management. Lots to be desired.
Looking forward to hearing how the experiments go.
