Edit, it seems that when I run the connman service in the container it takes control of the USB Wi-fi adaptor and I can’t no longer see it on the hostos at all.
Interesting. Do you see similar logs for openvpn?
Sure
The interesting thing is that I can access containers via the consoles, but containers updating, heartbeat and control of starting and stopping containers doesn’t work.
Dec 30 19:07:18 88887cb openvpn[3606]: Wed Dec 30 19:07:18 2020 library versions: OpenSSL 1.1.1b 26 Feb 2019, LZO 2.10
Dec 30 19:07:18 88887cb openvpn[3606]: Wed Dec 30 19:07:18 2020 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Dec 30 19:07:19 88887cb openvpn[3606]: Wed Dec 30 19:07:19 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]54.83.183.10:443
Dec 30 19:07:19 88887cb openvpn[3606]: Wed Dec 30 19:07:19 2020 Socket Buffers: R=[131072->131072] S=[16384->16384]
Dec 30 19:07:19 88887cb openvpn[3606]: Wed Dec 30 19:07:19 2020 Attempting to establish TCP connection with [AF_INET]54.83.183.10:443 [nonblock]
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 TCP connection established with [AF_INET]54.83.183.10:443
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 TCP_CLIENT link local: (not bound)
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 TCP_CLIENT link remote: [AF_INET]54.83.183.10:443
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 TLS: Initial packet from [AF_INET]54.83.183.10:443, sid=e223831d 03eafab3
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 VERIFY OK: depth=1, C=US, ST=WA, L=Seattle, O=balena.io, OU=balenaCloud, CN=open-balena-vpn-rootCA
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 VERIFY KU OK
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 Validating certificate extended key usage
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 VERIFY EKU OK
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 VERIFY OK: depth=0, C=US, ST=WA, L=Seattle, O=balena.io, OU=balenaCloud, CN=vpn.balena-staging.com
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 2048 bit RSA
Dec 30 19:07:20 88887cb openvpn[3606]: Wed Dec 30 19:07:20 2020 [vpn.balena-staging.com] Peer Connection Initiated with [AF_INET]54.83.183.10:443
Dec 30 19:07:21 88887cb openvpn[3606]: Wed Dec 30 19:07:21 2020 SENT CONTROL [vpn.balena-staging.com]: 'PUSH_REQUEST' (status=1)
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 PUSH: Received control message: 'PUSH_REPLY,sndbuf 0,rcvbuf 0,route 52.73.237.197,ping 10,ping-restart 60,socket-flags TCP_NODELAY,ifconfig 10.240.16.48 52.73.237.197,peer-id 0,cipher AES-256-GCM'
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 OPTIONS IMPORT: timers and/or timeouts modified
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 OPTIONS IMPORT: --sndbuf/--rcvbuf options modified
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 Socket Buffers: R=[131072->131072] S=[87040->87040]
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 OPTIONS IMPORT: --socket-flags option modified
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 Socket flags: TCP_NODELAY=1 succeeded
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 OPTIONS IMPORT: --ifconfig/up options modified
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 OPTIONS IMPORT: route options modified
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 OPTIONS IMPORT: peer-id set
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 OPTIONS IMPORT: adjusting link_mtu to 1627
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 OPTIONS IMPORT: data channel crypto options modified
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 Data Channel: using negotiated cipher 'AES-256-GCM'
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 ROUTE_GATEWAY 192.168.42.1/255.255.255.0 IFACE=eth0 HWADDR=f0:4c:d5:55:ce:d4
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 TUN/TAP device resin-vpn opened
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 TUN/TAP TX queue length set to 100
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 /sbin/ip link set dev resin-vpn up mtu 1500
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 /sbin/ip addr add dev resin-vpn local 10.240.16.48 peer 52.73.237.197
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 /etc/openvpn-misc/upscript.sh resin-vpn 1500 1555 10.240.16.48 52.73.237.197 init
Dec 30 19:07:22 88887cb openvpn[3606]: resin-ntp-config: Found config.json in /mnt/boot/config.json .
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 /sbin/ip route add 52.73.237.197/32 via 52.73.237.197
Dec 30 19:07:22 88887cb openvpn[3606]: ip: RTNETLINK answers: File exists
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 ERROR: Linux route add command failed: external program exited with error status: 2
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 GID set to openvpn
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 UID set to openvpn
Dec 30 19:07:22 88887cb openvpn[3606]: Wed Dec 30 19:07:22 2020 Initialization Sequence Completed
Terminal access works through an openvpn tunnel, while the stuff that isn’t working is handled by the supervisor. I have pinged the supervisor’s maintainers as it looks like an issue with the it specifically. We’ll update you when we have something more concrete.
1 Like
Actually, the issue with the supervisor might be related to https://github.com/moby/moby/issues/32106. I pinged the engine’s maintainer, but in the meantime could you try the solution in that GitHub issue? On a balena device you’d do something like this through the hostOS terminal:
$ mount -o remount,rw /
$ cat /etc/resolv.conf
......
$ nano /etc/docker/daemon.json
# Add DNS addresses
$ systemctl restart balena
The output from resolv.conf is 127.0.0.2,
Is it that I need to add this address to docker’s daemon, or add a known good DNS server to that? (E.g 8.8.8.8)
Thanks
8.8.8.8 should work
Hmm, there isn’t a file at /etc/docker/daemon.json?
You can create it
I’m guessing however that docker on the host os is using a different configuration file?
Having a look at https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file there’s a significant amount of config that’s essentially missing?
At a bare minimum I’d guess the following?
{
"dns": ["8.8.8.8"]
}
Also, Nano isn’t installed on BalenaOS 
Ok that didn’t work at all.
Upon some investigation the configuration file for balena-engine is /etc/balena-engine/daemon.json and contains the same contents but unfortunately it made no difference. I guess it auto copied on boot.
Edit, here’s a log file:
Dec 30 21:37:44 88887cb balenad[2719]: unable to configure the Docker daemon with file /etc/balena-engine/daemon.json: the following directives are specified both as a flag and in the configuration file: dns: (from flag: [10.114.102.1], from file: [8.8.8.8])
It seems that a DNS server of 10.114.102.1 is being set elsewhere?
Ok so:
I have found I can run the connman service in the container to only take control of a specific interface (wifi in this case).
This then prevents the host from experiencing issues, it seems for some reason when connman runs it breaks dns resolutions.
However I then experience DNS breaking inside just the container running connman.
Hi Ryan
In terms of specifying DNS servers, you can have a look at this section in the documentation, which provides guidelines on modifying the dnsServers depending on whether you’re willing to reprovision the device, or want to make the edit on an already provisioned device.
Kind regards
Alida
That won’t resolve this issue.
Hi Ryan, the issue is probably CM taking over the interface and breaking internal DNS resolution on the docker/balena bridge. Perhaps it’s also changing the default gateway. Without physical/local access to the device, it’s rather difficult to troubleshoot further remotely. If you can somehow prevent CM from messing up the outgoing interface and docker bridge DNS resolution (multiple WiFi cards?), you may be able to get this to work. Though in our experience, these solutions tend to be brittle.
1 Like
Hi @mpous,
I wasn’t able to resolve this, I think the issue is that it seems networkmanager and connman just do not play alongside eachother well at all.
So we can use balena I’m instead re-writing the software entirely to use networkmanager instead
Thanks for the help.