This might be more a docker compose question, but relates to how DNS configuration is inherited to containers when running in BalenaOS:
When we run several docker containers that talk to each other over gRPC, we found that the name-resolution right after a device has started up, can take up to a 30 seconds per connection before it is functioning. This surprises us especially because the docker containers should be able to resolve names of other containers via the Docker Deamon DNS and that -should- be incredibly quick, but apparently is not in case there is no internet connection.
What we tested:
- Have a balenaOS system with a WIFI router connected, WIFI Router not having a connection to the internet
- Deploy several containers from a defined docker-compose, and some of these containers interconnect via gRPC URLs
- Config of BalenaOS itself is:
root@balenasystem-123:~# cat /etc/resolv.conf
# we use dnsmasq at 127.0.0.2 so that user containers can run their own dns cache and forwarder and not conflict with dnsmasq on the host
nameserver 127.0.0.2
options timeout:15
We found that when we configured the gRPC URLS for the docker containers with a “passthrough://” parameter (example `passthrough://containername1:9090`, the connection (in golang: grpc.NewClient(url…)) call, goes quick, but if we do not have that passthrough, there is going to be a DNS lookup executed which takes way too much time (there were several connections to be made, maybe 2 tries for 2 servers adding up to a full minute).
Questions:
- What makes the containers try to use DNS from outside of the docker network?
- What could cause BalenaOS to run the containers and provide the containers with some DNS host attributes, maybe this is a default of running dockers in balena?
- How can we deploy the containers in a way that only docker daemon network resolution takes place? That is, docker names in the url should be immediately found and resolved by docker daemon, without any ‘host DNS lookup’ or even forwarding that to ‘external WIFI router DNS lookup’