Container network issue with 2.89.15 and cp-zookeeper/Redhat ubi8 containers

When testing the new 2.89.15 release of Balena I came across a container networking issue: I can’t ping certain containers by their name, although DNS resolution and pinging by IP work fine.

We’re using a multi-container setup in our fleet of UpBoards running balena. Some of these containers run a kafka stack, e.g. the cp-zookeeper image.
I noticed that the confluent containers couldn’t connect to each other, which I could reproduce by not being able to ping the confluent containers by their name. Apparently this works fine on the old 2.68 Balena version.

Test setup
I simplified the issue but looking up the base image used (Redhat ubi8) and was able to generate a simple docker-compose.yml to reproduce this issue:

version: "2.1"

    image: balenalib/intel-nuc-ubuntu:bionic
    entrypoint: ["tail", "-f", "/dev/null"]
    container_name: test1
    image: balenalib/intel-nuc-ubuntu:bionic
    entrypoint: ["tail", "-f", "/dev/null"]
    container_name: test2
    entrypoint: ["tail", "-f", "/dev/null"]
    container_name: zookeeper

I can ping the containers test1 and test2 from each other but I cannot ping zookeeper by hostname. The DNS seems to be resolved correctly and ping by IP works, however ping by hostname doesn’t (only get report of one package after stopping it with ctrl+c).

Container test1:

root@37df5e0c5fe5:/# nslookup zookeeper

Non-authoritative answer:
Name: zookeeper

root@37df5e0c5fe5:/# ping zookeeper
PING zookeeper ( 56(84) bytes of data.
^C64 bytes from icmp_seq=1 ttl=64 time=0.229 ms

— zookeeper ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.229/0.229/0.229/0.000 ms

root@37df5e0c5fe5:/# ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=0.259 ms
64 bytes from icmp_seq=2 ttl=64 time=0.196 ms

Notice the “^C” before the printout in the second command. I’ve never seen this behavior of the ping command before.

I’m looking for advice on how to pinpoint the issue. If useful I can upload further information (balena inspect of the container or so).

Hi @hesch,

Thanks for all the info. It would definitely be helpful to include some info from inspecting the containers, as well as inspect output of the Docker networks on the device. By default, if the network is not specified, the Supervisor on the device will add all the containers to a managed bridge network, thus avoiding the Docker issue of containers not being pingable with a default bridge network. With managed bridge network, the containers should be able to ping each other if they’re on the same subnet and don’t have conflicting IPs - you’ll be able to see this when you inspect the containers. Normally with Docker networks, there shouldn’t be conflicting IPs, but we’ve seen instances of IP conflicts possibly caused by unclean engine cleanup. For example: Port already in use, because proxy keeps binding to the wrong container IP · Issue #272 · balena-os/balena-engine · GitHub, although I don’t know if this is related.


just checking to see if you have managed to sort this out or you’d like some more help.

Hi and thank you for your replies. Unfortunately the topic got delayed quite a bit by more urgent things coming up. I’ll reconnect the device and provide more info from the inspect.

@cywang117 @ramirogm
First of all sorry for the big delay in providing information.

I set up a test device with the simple docker-compose.yml for testing posted above. Below you find the output of the inspect of the containers and the network. We were able to reproduce the issue on a different device with the Intel-NUC image as well.

# balena network ls
NETWORK ID          NAME                DRIVER              SCOPE
dfbc2d020fa0        1909342_default     bridge              local
c4c32ed6b321        bridge              bridge              local
e77ddd72d7c1        host                host                local
6d692447e105        none                null                local
3d1ae74b18fc        supervisor0         bridge              local
# balena network inspect dfbc2d020fa0
        "Name": "1909342_default",
        "Id": "dfbc2d020fa06fcc131ad09b9a9814aaca789aa5d2cc78a0a96f4b4e6ae496e6",
        "Created": "2022-09-01T14:11:39.449383485Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                    "Subnet": "",
                    "Gateway": ""
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        "ConfigOnly": false,
        "Containers": {
            "68bc0c9022b206aae7dd6a8c967f974fcf958a55349504b41fd31ce52d446960": {
                "Name": "test1_4595790_2080410_6e749a05e5e48b387d3d3bd89fced5ce3d7075d4",
                "EndpointID": "2de96235d50e8ef205cc2a5d2bfdd34014ed515b20a1761b57fb3b8fe0d812d2",
                "MacAddress": "02:42:ac:11:00:04",
                "IPv4Address": "",
                "IPv6Address": ""
            "7f0c5e01da18e5b339949306ea1d6bc48e99f5b44aa6ef7530b702889eaf8139": {
                "Name": "test2_4595791_2080410_6e749a05e5e48b387d3d3bd89fced5ce3d7075d4",
                "EndpointID": "4d8597df17bdee90283a093d50b90c36c387f38c53c656f950ebae3e3a600bb6",
                "MacAddress": "02:42:ac:11:00:03",
                "IPv4Address": "",
                "IPv6Address": ""
            "86466190381c690ce5dde54a9f380ed92fa8c5c1feeef1fc513000354fede2aa": {
                "Name": "zookeeper_4595792_2080410_6e749a05e5e48b387d3d3bd89fced5ce3d7075d4",
                "EndpointID": "52187b9e0feca687b34b9d0f01842c623ddb53cd2aed24acefbcf4af815f2a45",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "",
                "IPv6Address": ""
        "Options": {},
        "Labels": {
            "io.balena.supervised": "true"

I attached the full results of inspect on the containers here:
test1_inspect.txt (10.6 KB)
test2_inspect.txt (10.7 KB)
zookeeper_inspect.txt (11.5 KB)

If desired I can grant support access.