Problem:
I run into a problem where I am able to ping within the Host OS, but I am unable to do this from within the container.
In the container, pinging for the wwan0 device resolves in a connect: Network is unreachable message. The same configuration is working for other devices running exactly the same container and hardware setup.
We can reproduce this with ping -I wwan0 google.com on the Host OS (works) and within the container (does not work). We canping -I wwan0 8.8.8.8 from within the container which indicates that we might be dealing with a DNS problem.
Device:
Raspberry Pi 3
Huawei e3372 (configured with a file in /mnt/boot/system-connections/)
Together with the 4G network connection, we also have a working ethernet connection
Debugging information Within the HostOS
ip route
default via 192.168.178.1 dev eth0 metric 100
default via 100.127.237.121 dev wwan0 metric 700
10.114.101.0/24 dev balena0 scope link src 10.114.101.1
10.114.102.0/24 dev resin-dns scope link src 10.114.102.1
10.114.104.0/25 dev supervisor0 scope link src 10.114.104.1
52.4.252.97 dev resin-vpn scope link src 100.64.4.124
100.127.237.120/30 dev wwan0 scope link src 100.127.237.122 metric 700
172.17.0.0/16 dev br-2a1585066258 scope link src 172.17.0.1
192.168.178.0/24 dev eth0 scope link src 192.168.178.128 metric 100
ip route
default via 192.168.178.1 dev eth0 proto dhcp metric 100
default via 100.127.237.121 dev wwan0 proto static metric 700
10.114.101.0/24 dev balena0 proto kernel scope link src 10.114.101.1 linkdown
10.114.102.0/24 dev resin-dns proto kernel scope link src 10.114.102.1
10.114.104.0/25 dev supervisor0 proto kernel scope link src 10.114.104.1 linkdown
52.4.252.97 dev resin-vpn proto kernel scope link src 100.64.4.124
100.127.237.120/30 dev wwan0 proto kernel scope link src 100.127.237.122 metric 700
172.17.0.0/16 dev br-2a1585066258 proto kernel scope link src 172.17.0.1 linkdown
192.168.178.0/24 dev eth0 proto kernel scope link src 192.168.178.128 metric 100
Could you please clarify how you configure your container? Do you deploy it with a docker-compose.yml file (as described in our docs https://www.balena.io/docs/learn/develop/multicontainer/) or without? If you do this with docker-compose.yml file, do you set any value to network_mode?
“Network is unreachable” error may be caused by the fact your container is in a dedicated network created by the engine which is isolated from the host.
A short summary of possible container network modes can be found here:
Thank you for your response! We deploy with a normal Dockerfile, so it is basically a single container running on the device. We don’t use a docker-compose.yml file.
Ok, got it.
You mentioned that you have a working ethernet connection. Does it mean that when you do ping -I <ethernetInterface> google.com, it succeeds?
Ok, I see. And how do you get the DNS server name? Is it set by the DHCP server or is it a static config in system-connections? I would suggest trying to ping your DNS server via wwan0 interface. Is it possible the DNS server you get configured is unreachable from the IP you get with the modem?
The DNS server is set to DHCP (method=auto). Trying to ping the DNS server (IP from /etc/resolv.conf) ping -I wwan0 10.114.102.1 does not yield a response. I tried changing the DNS to 8.8.8.8 and service networking restart, but it looks like that does not make a difference as well.
To make the problem a bit stranger, I did some additional tests. It looks like the problem depends on the domain we are trying to ping
ping -I wwan0 google.com --> connect: Network is unreachable
ping -I wwan0 balena.io -> PING balena.io (52.72.230.122) from 100.127.237.122 wwan0: 56(84) bytes of data. Looks like the hostname was resolved, but no response
ping -I wwan0 reddit.com -> PING reddit.com (151.101.1.140) from 100.127.237.122 wwan0: 56(84) bytes of data.
64 bytes from 151.101.1.140 (151.101.1.140): icmp_seq=1 ttl=57 time=41.2 ms We get a response
Oh and these tests were done from within the container.
@larsjeh I think we should exclude the DNS server issues… If you deployed with docker-compose.yml file, you could set a DNS server to 8.8.8.8 for the container as described here:
Hi @larsjeh !
Just wanted to check with you if you still have this issue. I thought if the problem manifests itself only for particular addresses, maybe it was an intermittent issue on the cellular network provider side, so they could not route your packets properly?
Most possibly some TCP header parameters that are different in the packets sent from the host OS and the containers are causing the modem to trip. To be able to narrow down the issue a packet capture needs to be run and packets to be checked with wireshark. If you still experience the issue I may help out with running the diagnostics for you.
Some other important aspects are to run latest firmware on the modem, and also make sure you run the latest balenaOS released as it contains an updated ModemManager version.
