Help exposing container port

Product support balenaOS
, ,
1

Hello! I have just started playing around with Balena and an openBalena server. Currently, I have an application setup that is running a front end and Home Assistant deployed onto an RPi 3. From my knowledge, I have set up the correct exposed ports and should be able to connect into the dashboard that Home Assistant serves by default.

That, unfortunately, isn’t the case. It appears the port is exposed correctly however I cannot connect.

Here is my docker-compose.yml

version: "2"

volumes:
  volume_homeassistant:

services:
  react:
    build: ./react
    privileged: true
    restart: always
  homeassistant:
    build: ./homeassistant
    volumes:
      - volume_homeassistant:/config
    privileged: true
    restart: always
    ports:
      - "8123:8123"

And the Home Assistants Dockerfile

FROM homeassistant/raspberrypi3-homeassistant:latest

COPY config /imported

CMD cp -a /imported/. /config && python -m homeassistant --config /config

I have also set up port forwarding using iptables as described in the FAQ

iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 8123 -j REDIRECT --to-port 8123

Does anyone have any suggestions that will allow me to connect on this port?

2

Hey, thank you for the very detailed report! This really helps set the context. My first question would be, what manner of error are you seeing on connecting? Is it simply refusing to TCP connect? And what manner of network is the device resident in?

3

Thanks for the quick reply it is much appreciated!

When I attempt to connect using a browser I simply get an empty response. Performing a port scan also doesn’t show port 8123 being open only TCP ports 2375 and 22222.

As for the manner of network, I’m assuming you mean what type of network is the device deployed in? If that is the case it is a pretty standard small corporate network. We have an internal DHCP and DNS server with a Dell Firewall. Are you looking for any specifics?

4

Ah, are you connecting from within the network, as well? It may be a firewall issue if you’re connecting from outside (or, does the firewall operate even between hosts in the corp net?). The port expose stanza in docker-compose.yml is correct.

5

Everything is connected on the same internal network. Our firewall doesn’t have any host to host rules on this private subnet. If I spin up another Home Assistant instance running inside Docker on my Mac I am able to connect to it from another device as expected.

Is the iptables rule from the FAQ requireed or should I be able to see the port as soon as it is exposed in the docker-compose.yml?

8

Hi and sorry for the delay.
You should be able to see the port as soon as it is exposed in the docker-compose.yml, but I believe you’ll need to set the network to host in order to be able to do so

9

And generally - yes, any requirements you see in the FAQ/docs is required for the related use-case(s)

11

How do I specifically set the network to host?

From the documentation and it is more detailed within this masterclass when working with multi-container applications the balenaEngine automatically creates a bridge network and will only allow ingress traffic from containers that listen on a specific port. After reading that I feel that is what I have done in my docker-compose.yml.

12

Yes, specifically this step is the one that opens the port, but as you can see in the specific docker-compose.yml file of the frontend of the application, the network_mode of the frontend service is set to host.
E.g.

version: '2.1'
services:
  frontend:
    build: ./frontend
    network_mode: host
13

I tested running just the HA service with network mode in host however the results are the same. With that being said that leads me to believe the issue is on the host networks side of things. I did add that one iptables nat rule however am not very familiar with iptables.

This is an output of all the current tables active rules. As I attempt to hit this endpoint the packet counts increase which means it is getting dropped here somewhere. Would you agree?

Chain INPUT (policy ACCEPT 777 packets, 94329 bytes)
 pkts bytes target     prot opt in     out     source               destination
  282 19272 ACCEPT     tcp  --  lo     *       0.0.0.0/0            0.0.0.0/0            tcp dpt:48484
   10   914 ACCEPT     tcp  --  resin-vpn *       0.0.0.0/0            0.0.0.0/0            tcp dpt:48484
    3   192 REJECT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:48484 reject-with icmp-port-unreachable

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  210 10920 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
  210 10920 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
  105  6720 DOCKER     all  --  *      br-4512bc0a6f11  0.0.0.0/0            0.0.0.0/0
  105  4200 ACCEPT     all  --  br-4512bc0a6f11 !br-4512bc0a6f11  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 325 packets, 44340 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (3 references)
 pkts bytes target     prot opt in     out     source               destination
   17  1088 ACCEPT     tcp  --  !br-4512bc0a6f11 br-4512bc0a6f11  0.0.0.0/0            172.17.0.3           tcp dpt:8123

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
  105  4200 DOCKER-ISOLATION-STAGE-2  all  --  br-4512bc0a6f11 !br-4512bc0a6f11  0.0.0.0/0            0.0.0.0/0
  210 10920 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (3 references)
 pkts bytes target     prot opt in     out     source               destination
  105  4200 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
  210 10920 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
15

Hi there,

I’ve been asked to comment in here by my colleagues. Unfortunately, I think there’s been some misunderstanding here. You should not have to use host networking at all, as this binds a service to all host interfaces, which is usually not want you want in a multicontainer application.

The use of the ports property in your docker-compose.yml manifest is enough to open up the ports required for traffic ingress into the services it’s used on. In the case of your manifest, you’ve opened up port 8123 to map to port 8123 of traffic coming into your host. You should not start setting up any extra iptables rules. The documentation in our FAQ is slightly out of date, and this precedes multicontainer applications (single service/container applications in balena always use host networking). By default, your services in a multicontainer application are networked on a bridge which only allows internal traffic flow unless the ports keyword is used. As you have used it, you should be able to communicate with the services on port 8123.

When you say when you connect you get an ‘empty response’, what do you mean by this? If the port wasn’t opened, you’d see a ‘connection refused’ or similar error. For example:

curl -vv http://<hostname>:8123

(I’ve guessed a protocol here as HTTP, I don’t know what your service should actually be communicating in, but it will at least show up a connection refused error if there is one).

I note you’ve also got a service running something called react which vaguely suggests to me this is a React service running a UI? As there are no opened ports for this service, only traffic between the homeassistant and react service internal to the bridge will be available, and not to an external host.

Additionally, what kind of balena device is this, and are you trying on balenaCloud or on openBalena itself? If you’re new to balena, I’d strongly recommend starting on balenaCloud and getting to know the basics there first before launching into openBalena.

Have you followed the bridged networking example in the Services Masterclass to ensure that this works on your device? Alternatively, there’s a comprehensive Multicontainer Getting Started project here, which allows ingress of traffic into port 80 from the host to get live statistics on the device it’s installed on.

Best regards,

Heds

1 Like
18

Back to those ports, you’ve written for specific, but find completely different ones open.

Which port scanning tool did you use? They all beat up a network “differently” Currently for hardcore scanning, my go to tool has been Wireshark as SU for over 20 years. Else, Zenmap as SU probes differently, and had results that need more subjective interpretation, IMHO.

19

The ports open on the balena device in this case are due to the use of a development device, where port 22222 is for the use of SSH and 2375 is the balenaEngine socket for remote connection (and local balena pushes). I’d suggest in the current case that the change of iptables rules manually has resulted in the problem seen, so that 8123 isn’t visible (hence checking the other projects mentioned and ensuring they run correctly on the relevant device).

All portscanners use the same methodology of “beating up” a network, to lit they attempt to make connections to an IP address on a range of ports, both via TCP and UDP, and catalogue any port which does not refuse a connection (although the latter is more tricky for obvious reasons and usually results in an ICMP unreachable message should an appropriate port not be available, that said some scanners also attempt layer 7 connections for app specific traffic). Any comment to the contrary is incorrect.

Wireshark does not carry out port scanning, it uses libpcap to put one or more interfaces into promiscuous mode (as does tcpdump and almost any other network analysis tool for Linux, macOS or Windows) to show you traffic flow on those interfaces.

1 Like
21

The detail in your response is much appreciated Heds! Killer support.

After playing with the iptables for a bit without luck I decided to try a full fresh image. Downloaded the latest stock image for my RPi, configured it for the openBalena server, and pushed my testing application. After everything pulled down I was stuck with the same result.

Everything on the Balena end of things was setup correctly as you confirmed so I figured I’d take a deeper look into the service itself. I should have done that sooner. It turned out that my HA configuration file wasn’t being pulled in correctly which caused the system to not start fully. All is working now.

Thank you all for your help!