Reaching other container with network_mode host

Product support balenaOS
#1

Hi all,

I have one container that’s using network_mode: host. This is necessary, because I use dnsmasq inside that container for the Wi-Fi chip to function as access point. This needs direct access to wlan0. I’d prefer to not use network_mode: host at all for this container, but at this time it’s the only thing that works.

Having said that, I’d like for that container to access another container (my Redis service). Normally, I’d just have to connect to the name of the container, so redis in this case. But this doesn’t work, because the redis container uses the bridge network and my other container uses the host network.

Is there a way to connect (reliable) to the redis container, without exposing the redis container to the network of the device? I don’t want to have direct access to that container from outside the device for security reasons. So simply adding network_mode: host to that container doesn’t seem like a great idea.

Thanks in advance!

#18

Hi, sorry for the late response on this.

So you would like to put a container in host network mode and be able to access a redis instance running in a container that is using bridge network mode. The solution is to define ports in the docker-compose.yml file for the redis container and expose the redis port to the host’s network namespace.

Once you map the internal redis port like that you may also consider applying some iptables firewall rules to deny access to this port from the external network/Internet (e.g. over the eth0/wlan0 interfaces). You may apply those from the container running in host network mode.

If you have the opposite use-case where you would like to reach a container with host network mode from a container using bridge network mode - this is also possible. The application running in the host network container needs to listen either on the 172.17.0.1 address (the br-xxxxxxxxxxxx interface), or on all addresses (0.0.0.0). Then you can reach this application from the bridged container through the 172.17.0.1 address, which happens to be the gateway of that container. If you choose to run the application over 0.0.0.0 you may again consider using iptables to filter out access through eth0 or wlan0.

1 Like
#22

Hi Zahari,

Thanks for your answer!
I’m not very familiar with iptables. I know what it does, but I need to dive deeper in the rule creation.

My solution, for now, is also expose the redis container to the host network, but bind it to 127.0.0.1. This will also fix the issue that Redis is exposed to the outside world.

I’ve stumbled upon something else at the moment, which is exposing a container port, but not to the whole network. So, for example, I have a web server running on port 80. If I connect to it via the local IP address, let’s say 192.168.1.2, I want it to reject the request. However, if I tunnel port 80 via balena tunnel <device> -p 80:80, and so act as if I’m on the same network, I’d like it to accept that connection.

This is needed for debugging purposes on production devices. I don’t want that the web server running on the device is exposed, only when I’m tunneling to it.

Is this something that can be done?

Thanks in advance!

#23

I do not have direct experience with this, but you may try exposing a port to localhost instead like this:

ports:
  - "127.0.0.1:80:80"

If that does not work, then this could be solved through iptables filtering. The reason is that normal traffic with come either from eth0 or wlan0, where the tunneling one should be flowing through the VPN interface. The localhost approach is more simple and probably will work.

#24

I’ve tried it, and it seems to work on the host OS (curl 127.0.0.1 gives a good response).
However, when trying via the VPN tunnel (balena tunnel <device> -p 80:8081), I get some errors:

image
image776×93 15.2 KB

And the browser gives an ERR_EMPTY_RESPONSE.

Any idea’s?

Thanks so far anyway!

#25

Hi there, can you please share you docker-compose file which describes all of these services and port mappings, so that we can help you work out what is failing and why.

#27

Hi Anton,

I’m happy to share the docker-compose file. I’d just like to express that this seconds question has nothing to do with the network_mode: host anymore. This is fixed by using the bind 127.0.0.1 in the config of Redis.

I just want to have the HTTP webserver available when I tunnel the device’s port using balena tunnel, but not when I access it directly via it’s local IP (e.g. via 192.168.1.2).

The docker-compose.yml:

services:
  webserver:
    build: ./webserver
    restart: always
    ports:
      - "127.0.0.1:80:80"

This way, I can access the webserver from within the host machine. So I SSH into the host and, without entering the container, I can curl to 127.0.0.1, which gives me a fine response. I can’t access it via the local IP, because it’s bound to 127.0.0.1. However, if I tunnel port 80 via balena tunnel, I get the errors as shown in my previous post. I know it has something to do with the binding of 127.0.0.1 to port 80, but I’d like to know if there’s a way to access it via it’s localhost & via the tunnel, but not via the local IP (for security reasons).

Thanks in advance!

#30

Hi Bart,

I have not been able to reproduce this. That is I am able to restrict access by binding to 127.0.0.1 and the webpage doesn’t show when I try to access the device ip directly.
I can however stilll acess it by ssh’ing into the device and using curl 127.0.0.1. I can also access the page if I use balena tunnel

I used balena tunnel <uuid> -p 80:8081 and was able to access the webpage on my host computer at 127.0.0.1:8081

Can you please share your docker-compose file?

#32

Hi Rahul,

I’ve added the docker-compose.yml in my previous post. The only difference is that I can’t reach it via the balena tunnel command.

This is my command:

balena tunnel <device> -p 80:8081

And I’m getting 500 errors. If it’s any difference, I’m using Node.js with Express and as hostname 0.0.0.0, so it binds itself to everywhere.

Thanks!

#33

Hey Bart,

I also have a nodejs app running on port 80 and have been able to access it via balena tunnel

Maybe you can provide support access and we can have a further look? you will need to share the device uuid. thanks

#34

Hi Rahul,

I’d be happy to share the UUID to provide support access, only thing is, I’m using openBalena. So it probably has no use for you, right?
Or is it something of openBalena why it doesn’t work? Tunneling port 22222 and ssh’ing into the device works like a charm, so I don’t think it’s that.

For my knowledge, you’ve also started a Node.js app, running a webserver on port 80 and used 127.0.0.1:80:80 in your docker-compose.yml?

I keep getting the 500 error and in my browser a ERR_EMPTY_RESPONSE…

#36

Let me retry

#37

Hey there

I was able to reproduce this with

version: '2'
services:
  main:
    build: ./
    restart: always
    ports:
      - "127.0.0.1:80:80"

I had to set the service to network_mode: host to be able to access the webpage via balena tunnel

Does this help?

#39

You’re right, I’m not using network_mode: host for this container. I don’t really want to expose more than I have to, and I don’t really need the host network mode (except for this). So I don’t know if I can use a cap_add for this situation or something else that’ll do the trick that you know of?

#40

Hi

I think you’ll need the network mode host here because the tunnel is something that would go through the host OS networking. Otherwise you’d need something that goes from the balenaOS to the bridge network for the containers, and finds the right service.

#41

Is there something specific that you are worried about wrt the host network mode?

#43

Hi Anuj,

There isn’t something I’m really worried about, but I’m always trying to keep the containers separate from the hostOS as much as possible. So only give them the capabilities and permissions they really need. If there isn’t another way for this, then I’ll apply network_mode: host. It just makes the container more ‘vulnerable’ and, in theory, it can screw things up.

But nevertheless, thanks for explaining in short why it doesn’t work like I thought it should with the VPN tunnel!

#44

Hi Bart

What I mentioned is based on the best of my understanding, but I have also pinged one of my colleagues. In case it is indeed possible, he’ll drop in a reply on this thread.

1 Like