I’m looking to migrate a project over to open-balena , but currently we have nginx-proxy running on 80/443 that automatically detects (server-side) docker containers and will redirect requests based on VIRTUAL_HOST=app-A.domain.com and VIRTUAL_PORT=8888 environment variables defined inside each container (and handles letsencrypt too).
Does open-balena handle any deployments of server-side containers (e.g. dashboards, data ingestion)? If so, it would be great if the built-in haproxy were able to automatically register launched containers and redirect requests on port 80/443 to custom urls. You could do this using docker-gen (like nginx-proxy does) to re-generate haproxy.cfg then SIGHUP when a new container is created. You could do something like BALENA_PROXY_HOST / BALENA_PROXY_PORT .
If that’s not an intended usage and open-balena is meant to be more limited in scope, then it would be useful to be able to run a balena server alongside containers that do cover this usage, meaning that I’d like to find a way to either redirect requests through nginx-proxy to haproxy or the other way around, but I’m not sure if that’s possible.
Personally, my preference is for the first option because it doesn’t involve having to run and manage two web servers. But that would also mean handling automatic LetsEncrypt (which I think you already do for the other endpoints). Side note, just because it may affect things like certificates, I’d like to have the ability to have api.balena.domain.com for the balena-specific endpoints and (e.g.) elasticsearch.domain.com so that the apps don’t have to be subdomains of *.balena .
I’m wondering what your thoughts are on this and how it fits into the roadmap of open-balena
Thanks for reaching out, it’s always nice to get feedback from the community. I can provide some insight into the ingress story with openBalena which might help you moving forward.
The HAProxy service in the openBalena stack is configured to route to the service containers based on the HTTP Host header value, with the exception of the VPN. The VPN setup is such that we listen on port 443 and then in HAProxy we route any non-TLS packets on port 443 to the VPN. This concept breaks if you put oB behind any kind of HTTP/TLS proxy since it cannot route the VPN traffic.
oB is designed to run on a single machine and take up all the resources there, so we didn’t consider the need for extra services when we put the ingress story together. This isn’t an oversight, but rather not something we every really intended to support as such. If you wanted to add functionality to the HAProxy container to also route your other traffic then I don’t see that being a problem, but it would be up to you to qualify it and support it.
There is a lot of conversation and work around our overall ingress story within balenaCloud and balenaOS so in the future we may change how it works in openBalena to account for these options. For now though, as long as you stick to the rules in the existing HAProxy container, then you can replace it with whatever implementation you see fit 
PS: It is possible, but a little hard to explain in a forum thread, to change which port the OpenVPN connection will use; so this could be a possibility it you wanted to replace HAProxy or change how it routes etc. You’d need to look at the API’s service environment variables, which are expanded out into the config provided at https://api.domain.com/os/v1/config. The devices use this endpoint to pickup their initial config when joining an instance.
Ah I see. This is very helpful, thank you.
So when you are designing a IoT server cluster, the intended design is to have your main cluster (say 1 manager and 2 workers) which runs the different dashboards and data ingestion servers and then have a dedicated open-balena control server? I guess I just hadn’t considered that as a design scenario, but I guess it’s workable. (I had just envisioned it running on the manager server)
In the long run, would you be opposed to a PR that provides haproxy the ability to register arbitrary docker containers behind its reverse proxy (basically nginx-proxy+docker-gen but with your haproxy config instead, no hard-coded services)? Or do you think it’s best to enforce having a separate dedicated balena server? I’d rather not make design choices that move away from the balena team’s vision because I’m going to have to hand this system off at some point and it wouldn’t be sustainable to manage a separate fork.
Hello Bea,
Thanks for your thoughts and insights. It would be best to start by creating an issue on https://github.com/balena-io/open-balena/issues that way maintainers can check it there and have a better discussion with you on the same.