Disabling NetworkManager DNS for shared connection

Product support
1

Hello,

I’m trying to run ResinOS on an RPi3 configured as a hotspot and router. The setup works well using the hotspot example from the docs, but now I’d like to set up pi-hole for network-wide ad-blocking. The recently-released pi-hole 4.0 ships an official Docker container, so it should definitely run in a container.

Unfortunately, it won’t launch because port 53 is in use. I did some poking around, and it looks like NetworkManager launches a dnsmasq instance for shared connections. To disable DNS in this instance and only use DHCP, you’d likely edit a file in /etc/NetworkManager/dnsmasq-shared.d and set port = 0.

I’ve spent a couple hours trying to figure out a way to do this. I understand that the root partition is read-only, and that files in the state partition are whitelisted. Is there some other way to disable DNS on the dnsmasq instance NetworkManager manages for shared connections so I can run a pi-hole container?

Note that I don’t want to change what DNS value my DHCP server returns. I’m fine with it returning what it does. I just want to run my own DNS server at that location.

I’ve also tried systemctl stop dnsmasq, but this only affects the upstream dnsmasq, not the one NetworkManager launches for shared connections. I also found a thread about running pi-hole, but it seems to imply that you can bind containers to the interface IP, which I don’t think is the case when NetworkManager is sharing the connection.

Thanks for any help.

4

WiFi Connect also spawns a dnsmasq instance in its container: https://github.com/resin-io/resin-wifi-connect/blob/master/src/dnsmasq.rs. It does not interfere with the one running in the host OS.

I used hostapd a few times as well and it needed a similar dnsmasq setup: https://github.com/resin-io-playground/hostapd-minimal

I have not used pi-hole, but I quickly saw now in its man page the following configuration option: -i, interface Specify dnsmasq's interface listening behavior. Will that be helpful?

6

I don’t think the issue is that Pihole needs a different interface configuration. The issue is that when using the AP configuration on the network setup page, with method=shared, NetworkManager binds port 53 in a way that Resin won’t let me change. So ultimately I need to pass -p 53:53 to my Docker command line, and that won’t work regardless of what happens on Pihole’s end because Docker is doing the binding.

I have, however, made a bit of progress. I put my wireless interface in manual mode and connected to it directly, manually setting an IP on my laptop. When doing this, I can ping out from the device, but it of course isn’t routing packets from my laptop. So I tried bridging my wifi and ethernet interfaces. The ethernet interface gets an IP automatically via DHCP, while I want the wifi interface acting as an access point. Unfortunately, the wifi interface never comes up. Here’s what I have. First, the bridge:

[connection]
id=br0
uuid=69614cbf-5245-449d-819d-2496ca7efade
type=bridge
autoconnect=true
interface-name=br0
permissions=

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

Not sure if all these autoconnect=trues are needed. Next, the ethernet. This gets an address from my provider via DHCP.

[connection]
id=bridge-slave-eth0
uuid=56793758-6a0b-4613-b17b-6d2efaac7a6d
type=ethernet
autoconnect=true
interface-name=eth0
master=br0
permissions=
slave-type=bridge

[ipv4]
method=auto

And wlan0, which I just want to assign a manual address and have it route packets. With this in place, port 53 should be unblocked and I can run Pihole:

[connection]
id=bridge-slave-wlan0
uuid=bb421627-971e-4637-a50b-4ffe61a33e73
type=wifi
autoconnect=true
interface-name=wlan0
master=br0
permissions=
slave-type=bridge

[wifi]
band=bg
mac-address-blacklist=
mac-address-randomization=0
mode=ap
seen-bssids=
ssid=ICanHasWifi

[wifi-security]
group=
key-mgmt=wpa-psk
pairwise=
proto=rsn
psk=xxx

[ipv4]
address1=192.168.0.1/24
dns-search=
method=manual

Any idea what I might be missing?

7

I also enabled persistent logging, plugged the Pi in for a couple minutes, then unplugged it. Nothing looked alarming in the review. All the interfaces appeared to be grought up, in the sense that NetworkManager said they were. But I never saw DHCP requests from the ethernet port, nor did my AP come up. I’m convinced I’ve configured this bridge wrong in a subtle way that I just can’t spot, but that is perfectly reasonable for NM to just bring up the devices and do nothing more. :slight_smile:
Thanks.

10

You were right about NetworkManager spawning a new dnsmasq instance when setting connection mode to shared. This runs on the same IP that you need pi-hole’s dnsmasq instance to run, so the port will not be available indeed. Manual option is the correct one for this case.

My suggestions would be:

  1. For connection sharing there are two possibilities NAT and bridged. I think you will be better off for your use case with NAT. This is also what NetworkManager does in its shared mode.

  2. Install pi-hole in a recent Debian container. Usually I go with either Stretch or even the unstable Buster if I have issues on Stretch.

  3. I looked at pi-hole’s installer. It checks for nmcli availability. nmcli is the command line interface of NetworkManager. It communicates with NetworkManager through D-Bus. You may install NetworkManager inside the container, so that the installer invokes nmcli. However do not forget to mask the NetworkManager’s service: RUN apt-get update && apt-get install -y network-manager && systemctl mask NetworkManager.service. You will need also to export the correct DBUS_SYSTEM_BUS_ADDRESS address, so that nmcli communicates with the NM service running on the host OS. More information on this here: https://docs.resin.io/reference/OS/network/2.x/#changing-the-network-at-runtime

  4. If you are running a multicontainer setup you need to set docker compose network_mode to host.

  5. Investigate what the pi-hole installer does, as it may well set networking for your use case. I have not done that, so I am not sure whether it will do that or not.

  6. Here is another topic related to Internet connection sharing that you may find useful which includes a sample repo from a forum user: Route Wireless to Wired with resinOS 2.x and RPi 3

  7. You may find useful the create_ap script, which is for creating Access Points with Internet sharing (it uses hostapd instead of the lightweight built-in access point NetworkManager creates, but it can still be useful for creating the AP, or as a reference at least): https://github.com/oblique/create_ap

Hope that is helpful :wink:

1 Like
12

Thanks for all the help. I’m almost there! My setup does almost exactly what I want, but because of the read-only rootfs, a couple things won’t persist. I’ll document everything here in case anyone googles how to set up pihole on ResinOS and needs pointers.

First, the hotspot config:

[connection]
id=resin-hotspot
uuid=36060c57-aebd-4ccf-aba4-ef75121b5f77
type=wifi
autoconnect=true
interface-name=wlan0
permissions=
secondaries=

[wifi]
band=bg
mac-address-blacklist=
mac-address-randomization=0
mode=ap
seen-bssids=
ssid=ICanHasWifi

[wifi-security]
group=
key-mgmt=wpa-psk
pairwise=
proto=rsn
psk=supersecretpasswordhere

[ipv4]
address1=192.168.0.1/24
dns-search=
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

That’s the only interface you need for my setup, which uses the Pi as my network router connected to a cablemodem which is assigned an IP via DHCP.

Next I assign 192.168.0.2/24 to my laptop’s network address, set the gateway to 192.168.0.1, and SSH to root@192.168.0.1:22222. I then run:

# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Now I set up a docker-compose.yml to launch pihole:

version: "2"

volumes:

  pihole:

  dnsmasq:

services:

  pihole:
    image: pihole/pihole:v4.0_armhf
    network_mode: host
    ports:
      - 53:53
      - 53:53/udp
      - 67:67/udp
      - 80:80
      - 443:443
    volumes:
      - pihole:/etc/pihole
      - dnsmasq:/etc/dnsmasq.d
    environment:
      ServerIP: 192.168.0.1
      WEBPASSWORD: super_secret_web_password
      DNSMASQ_LISTENING: local
    cap_add:
      - NET_ADMIN

Not sure if the NET_ADMIN is needed since I’ve set network_mode: host, but I just got this working and am hesitant to breathe on it wrong.

Before launching this, I need to disable dnsmasq:

# systemctl stop dnsmasq

Then, on my laptop:

$ DOCKER_API_VERSION=1.22 DOCKER_HOST=tcp://192.168.0.1:2375 docker-compose up -d

With this setup, you’re able to access http://pi.hole/admin/. You probably want to access Settings->DHCP, enable the DHCP server, set the range, etc. I think this sets up dnsmasq to listen on local interfaces, but couldn’t hurt to check under Settings->DNS.

Two problems with this setup. First, I need to set up the iptables rule on each boot. Next I need to either disable dnsmasq permanently, or just stop it again and again. Looks like the way to do this is to spin up other containers with the appropriate privileges to run these commands, and then make the pihole container depend on those starting up. I’ll try to integrate that later, but right now I’m going to just let this run and get back to work. :slight_smile:
Thanks a bunch for your help and pointers.

13

Nice work! You may also create an example GitHub repo whenever you are ready and have some additional time for this so that other Resin users may try it out :wink:

15

I hope it is ok to bring this thread back to life.

I am attempting to do something similar. I want to run a balena device that provides DNS service on the local network over port 53.

I have my DNS server container running fine when I bind the port’s container to something other than port 53 on the host. However I cannot bind port 0.0.0.0:53 because NetworkManager runs the dnsmasq service.

If I hard code the device’s ip address on the local LAN (e.g., 192.168.0.X:53) then of course it will work, because the NetworkManager’s dnsmasq is only binding locally.

However these devices will be used in unknown networks, so there’s no way to know the ip address of the device before hand.

AFAICT adding configs under system-connections will have no effect, because of the line dns=default in /etc/NetworkManager/NetworkManager.conf.

Is it possible to do the following?

  1. have eth0 configured via dhcp
  2. prevent NetworkManager from starting the dnsmasq service, allowing a container to bind to port 53.

For more context:

These devices will be deployed in a network where the router will auto assign a pre-chosen ip address (via dhcp) to the mac addresses of the devices. That way the device will have a “static” IP, but is otherwise plug-and-play. Wifi is not involved in my setup.

Did you ever manage to do this?

Ways I can think of to disable dnsmasq on a normal system:

  • systemctl disable dnsmasq.service
  • add bind-interfacesto /etc/dnsmasq.conf
  • set dns=none in /etc/NetworkManager/NetworkManager.conf

All these require writing to the root fs however.

18

Ok, using

mount -o remount,rw /
# edit  /etc/NetworkManager/NetworkManager.conf
# change dns=default -> dns=none
systemctl disable dnsmasq
systemctl mask dnsmasq
# add your own namservers to /etc/resolv.conf
reboot

Rebooted and it was persisted, and my container could bind to 53 on the host successfully.

So the question is how to edit the balena img to ship this change?

Mounting all the partitions in the image, I see the NetworkManager file lives at:

/mnt/loop2/balena/aufs/diff/4a9cd733c035c826172bf73ef15b4a17d61984fd0eb1fc27d762a307dedb20b3/etc/NetworkManager/NetworkManager.conf

Is editing this file sufficient? Will that change be blown away during a system upgrade?

Is there a way to leverage the resin-state partition to provide an override?

20

Hey @ramblurr,

I’ve dealt with this before and there’s actually a dnsmasq flag you can set in the official Pi-Hole container to avoid the binding issue you are seeing.
By adding bind-interfaces to /etc/dnsmasq.conf the instance will only bind to the interface you provide in your Pi-Hole DNSMASQ_LISTENING env var.

Here’s a link to the related line of code in my Pi-Hole repo running on balenaOS:

# force dnsmasq to bind only the interfaces it is listening on
# otherwise dnsmasq will fail to start since balena is using 53 on some interfaces
RUN echo "bind-interfaces" >> /etc/dnsmasq.conf

This way you can avoid changing the balena host OS and instead adjust how the Pi-Hole dnsmasq instance chooses which interfaces to bind to. Generally I would avoid modifying the host OS if at all possible as the changes are almost always lost on reboot with some exceptions.

Let me know how this works out!

Take care,
Kyle

1 Like
21

Hi Kyle,

Thanks for the info.

Unfortunately I’m not using pihole in this service, but rather coredns.

coredns is only able to bind to specific ip-addreses, not interfaces (see the docs).

I’m thinking now perhaps a solution is to run a script in the container that gets the ip address of the interface and injects this into the config.

22

I’ve done this now, it’s clean and works well. Thanks for the tip @klutchell. To share with the community, I’ve refactored the setup we’re using for this project into a standalone config suitable for a home or office network (similar to your pihole example), you can see that here: GitHub - Ramblurr/homelab-balena-coredns

23

Hey Casey, nice to hear that you got it working. Thanks for sharing your solution here. So cool.

1 Like
25

Hi, this thread looks very similar to my situation. But probably I did not understand the full details and I’d like to ask a question.

What I want to do:

  • to use my Manjaro Linux as Wi-Fi hotspot
  • to use my Manjaro Linux as dns server, using docker container qmcgaw/dns

Problem:

  • the port 53 is used by dnsmasq which is launched by NetworkManager.
  • if I set ‘method=shared’ at the hotspot configurations of the NetworkManager, the hotspot works with no problem but dnsmasq is launched using the port 53.
  • if I change the hotspot configurations as ‘method=manual’ or ‘method=auto’, the port 53 is free and my phone connects to the hotspot, but no internet after all.

My network interfaces:
enp2s0: ethernet interface of the Manjaro Linux
wlp3s0: wifi interface of the Manjaro Linux (inet 10.42.0.1)
docker0: docker host (inet 172.17.0.1)
bridge0: docker bridge (inet 172.19.0.1)
br0: bridge for wifi hotspot (currently no IP assigned)

My NetworkManager configurations:

  • bridge:

[connection]
id=Bridge connection 1
uuid=3dce9d65-7808-464b-aa5c-ff7352b3d708
type=bridge
autoconnect=true
interface-name=br0
permissions=

[bridge]

[ipv4]
dns-search=
method=auto

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

  • ethernet for br0:

[connection]
id=br0 slave 1
uuid=978d9e27-0c10-49e0-819b-7e30c1eabd54
type=ethernet
autoconnect=true
interface-name=enp2s0
master=br0
permissions=
slave-type=bridge

[ipv4]
dns-search=
method=auto

  • hotspot:

[connection]
type=wifi
autoconnect=true
interface-name=wlp3s0
bridge=br0
permissions=

[wifi]
mode=ap
ssid=myssid

[wifi-security]
key-mgmt=wpa-psk
psk=passwd
pairwise=ccmp

[ipv4]
address1=10.42.0.1/24
dns-search=
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

And I’ve done the command below:
sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Any idea what I might be missing?

26

Hey @wagnerian

From your description, I didn’t understand if you are using balenaOS. Are you running Manjaro on docker? Would it be possible to share your dockerfile?

28

Dear @anujdeshpande

I’m not using balenaOS. What I’m using is Manjaro Linux, and the docker container is Alpine-based DNS server. I’m not familiar with balenaOS, but I guess NetworkManager is the same both for balenaOS and Manjaro.

29

I faced this same problem as well, only on an Ubuntu Server instance but landed here in trying to find anyone else trying to accomplish this.

I was successful in my configuration - I’m not thrilled with what I had to do…but it’s working…open to fresh ideas of better ways.

  1. Rename the dnsmasq executable to something else by running: sudo mv /usr/sbin/dnsmasq /usr/sbin/dnsmasq.bak

  2. Create a new executable script in it’s place: sudo touch /usr/sbin/dnsmasq and sudo chmod +x dnsmasq

  3. Leave a breadcrumb for myself by making the contents of the script the following using sudo nano /usr/sbin/dnsmasq:

#!/bin/sh

echo "dnsmasq disabled and backed up as dnsmasq.bak"
  1. Reboot - now port 53 is assigned to pihole as represented by running sudo lsof -i :53