Unable to get Mosquitto operating on BalenaOS

I have a mosquitto container that is relaying messages to another container that is performing some calculations. No matter what configurations are applied the mosquitto container always falls over approximately every 6 minutes.

I have run the exact same configuration on the same hardware using Ubuntu and it runs without a problem, but as soon as BalenaOS is introduced the 6 minute failure appears. There are unfortunately no error messages to assist in the troubleshooting, but it does not seem to be mosquitto configuration related.

Underlying architecture: Amd64
Dockerfile Base Image utilised: balenalib/amd64-alpine:3.16-run

I have tried a few variations for the mosquitto Dockerfile. The latest variation is using the Dockerfile from the official eclipse-mosquitto page and replacing the base image with the balenalib equivalent to ensure both the balena environment and mosquitto requirements are catered for. This still runs well on ubuntu but not BalenaOS.

Official Mosquitto Dockerfile https://github.com/eclipse/mosquitto/blob/a8448a9c7b14bdaee6ec80419d43fd6544e789b6/docker/2.0/Dockerfile

Just for reference I did see that a mosquitto balenablock was created but I would prefer to use a simple Dockerfile for my mosquitto implementation.

Hello @Stephen welcome to the balena community!

Could you please share the docker-compose that you are using? I would like to test! And could you please share the logs when Mosquitto falls over?

Do you have any special configuration? Do you use any version in particular?


Hi Mark,

Thanks for your message.

You can utilise this for the docker-compose.yml

version: '2.4'


    driver: bridge


    build: ./mosquitto
    container_name: mqtt-broker
      - local-net  
    restart: always
      - 1883:1883
      - 9001:9001
      - mosquitto:/mosquitto/data
      - mosquitto:/mosquitto/log

  # simple_model:
  #   image: simple_model:latest
  #   container_name: simple_model
  #   networks:
  #     - local-net
  #   restart: always
  #   environment:
  #     - BROKER_ADDRESS=mqtt-broker
  #     - BROKER_PORT=1883
  #   volumes:
  #     - 'model:/data'

Here is the dockerfile:

FROM balenalib/amd64-alpine:3.16-run

LABEL maintainer="Roger Light <roger@atchoo.org>" \
    description="Eclipse Mosquitto MQTT Broker"

ENV VERSION=2.0.15 \
    DOWNLOAD_SHA256=4735b1d32e3f91c7a8896741d88a3022e89730a1ee897946decfa0df27039ac6 \
    GPG_KEYS=A0D6EEA1DCAE49A635A3B2F0779B22DFB3E717B7 \
    LWS_VERSION=4.2.1 \

RUN set -x && \
    apk --no-cache add --virtual build-deps \
        build-base \
        cmake \
        cjson-dev \
        gnupg \
        libressl-dev \
        linux-headers \
        util-linux-dev && \
    wget https://github.com/warmcat/libwebsockets/archive/v${LWS_VERSION}.tar.gz -O /tmp/lws.tar.gz && \
    echo "$LWS_SHA256  /tmp/lws.tar.gz" | sha256sum -c - && \
    mkdir -p /build/lws && \
    tar --strip=1 -xf /tmp/lws.tar.gz -C /build/lws && \
    rm /tmp/lws.tar.gz && \
    cd /build/lws && \
    cmake . \
        -DCMAKE_BUILD_TYPE=MinSizeRel \
        -DLWS_IPV6=ON \
        -DLWS_WITH_HTTP2=OFF \
        -DLWS_WITH_ZLIB=OFF && \
    make -j "$(nproc)" && \
    rm -rf /root/.cmake && \
    wget https://mosquitto.org/files/source/mosquitto-${VERSION}.tar.gz -O /tmp/mosq.tar.gz && \
    echo "$DOWNLOAD_SHA256  /tmp/mosq.tar.gz" | sha256sum -c - && \
    wget https://mosquitto.org/files/source/mosquitto-${VERSION}.tar.gz.asc -O /tmp/mosq.tar.gz.asc && \
    export GNUPGHOME="$(mktemp -d)" && \
    found=''; \
    for server in \
        hkps://keys.openpgp.org \
        hkp://keyserver.ubuntu.com:80 \
        pgp.mit.edu \
    ; do \
        echo "Fetching GPG key $GPG_KEYS from $server"; \
        gpg --keyserver "$server" --keyserver-options timeout=10 --recv-keys "$GPG_KEYS" && found=yes && break; \
    done; \
    test -z "$found" && echo >&2 "error: failed to fetch GPG key $GPG_KEYS" && exit 1; \
    gpg --batch --verify /tmp/mosq.tar.gz.asc /tmp/mosq.tar.gz && \
    gpgconf --kill all && \
    rm -rf "$GNUPGHOME" /tmp/mosq.tar.gz.asc && \
    mkdir -p /build/mosq && \
    tar --strip=1 -xf /tmp/mosq.tar.gz -C /build/mosq && \
    rm /tmp/mosq.tar.gz && \
    make -C /build/mosq -j "$(nproc)" \
        CFLAGS="-Wall -O2 -I/build/lws/include -I/build" \
        LDFLAGS="-L/build/lws/lib" \
        WITH_ADNS=no \
        WITH_DOCS=no \
        WITH_SRV=no \
        WITH_STRIP=yes \
        WITH_TLS_PSK=no \
        WITH_WEBSOCKETS=yes \
        prefix=/usr \
        binary && \
    addgroup -S -g 1883 mosquitto 2>/dev/null && \
    adduser -S -u 1883 -D -H -h /var/empty -s /sbin/nologin -G mosquitto -g mosquitto mosquitto 2>/dev/null && \
    mkdir -p /mosquitto/config /mosquitto/data /mosquitto/log && \
    install -d /usr/sbin/ && \
    install -s -m755 /build/mosq/client/mosquitto_pub /usr/bin/mosquitto_pub && \
    install -s -m755 /build/mosq/client/mosquitto_rr /usr/bin/mosquitto_rr && \
    install -s -m755 /build/mosq/client/mosquitto_sub /usr/bin/mosquitto_sub && \
    install -s -m644 /build/mosq/lib/libmosquitto.so.1 /usr/lib/libmosquitto.so.1 && \
    install -s -m755 /build/mosq/src/mosquitto /usr/sbin/mosquitto && \
    install -s -m755 /build/mosq/apps/mosquitto_ctrl/mosquitto_ctrl /usr/bin/mosquitto_ctrl && \
    install -s -m755 /build/mosq/apps/mosquitto_passwd/mosquitto_passwd /usr/bin/mosquitto_passwd && \
    install -s -m755 /build/mosq/plugins/dynamic-security/mosquitto_dynamic_security.so /usr/lib/mosquitto_dynamic_security.so && \
    install -m644 /build/mosq/mosquitto.conf /mosquitto/config/mosquitto.conf && \
    install -Dm644 /build/lws/LICENSE /usr/share/licenses/libwebsockets/LICENSE && \
    install -Dm644 /build/mosq/epl-v20 /usr/share/licenses/mosquitto/epl-v20 && \
    install -Dm644 /build/mosq/edl-v10 /usr/share/licenses/mosquitto/edl-v10 && \
    chown -R mosquitto:mosquitto /mosquitto && \
    apk --no-cache add \
        ca-certificates \
        cjson \
        libressl && \
    apk del build-deps && \
    rm -rf /build

VOLUME ["/mosquitto/data", "/mosquitto/log"]

RUN echo 'listener 1883' >> /mosquitto/config/mosquitto.conf
RUN echo 'allow_anonymous true' >> /mosquitto/config/mosquitto.conf
RUN echo 'persistence true' >> /mosquitto/config/mosquitto.conf
RUN echo 'persistence_location /mosquitto/data/' >> /mosquitto/config/mosquitto.conf
RUN echo 'pid_file /mosquitto/data/mosquitto.pid' >> /mosquitto/config/mosquitto.conf
RUN echo 'user mosquitto' >> /mosquitto/config/mosquitto.conf
RUN echo 'log_dest stdout' >> /mosquitto/config/mosquitto.conf
RUN echo 'log_dest file /mosquitto/log/mosquitto.log' >> /mosquitto/config/mosquitto.conf
RUN echo 'log_type all' >> /mosquitto/config/mosquitto.conf
RUN echo 'connection_messages true' >> /mosquitto/config/mosquitto.conf

RUN chown mosquitto: /mosquitto/config
USER mosquitto

CMD ["/usr/sbin/mosquitto", "-c", "/mosquitto/config/mosquitto.conf"]

@Stephen could you please share the logs when the mosquitto restarts?

On the other hand, did you try to build from the docker image from Eclipse mosquitto (e.g. below from balenAir) or do you need something specific?

    image: eclipse-mosquitto:1.6.15
      - "1883:1883"
    restart: always 

Hi @mpous, I actually never tested that, although thinking now I definitely should have.
I ran this for both version 1.6.15 and 2.0.15 with the same result being that they shut down just like before.

The logs are as follows:

mqtt  1674626640: Config loaded from /mosquitto/config/mosquitto.conf.
 mqtt  1674626640: Opening ipv4 listen socket on port 1883.
 mqtt  1674626640: Opening ipv6 listen socket on port 1883.
 mqtt  1674626640: mosquitto version 1.6.15 running
 mqtt  1674626861: mosquitto version 1.6.15 terminating
 mqtt  1674627130: mosquitto version 1.6.15 starting

I ran the health checks and the check container engine failed.
I have uploaded the device diagnostics check as well.

device_diagnostics.txt (1.3 MB)

Hi @mpous,

Do you know what causes this issue? Is there something to learn here or must I just reformat the device?

Kind regards,

Looks like the supervisor or the balenaEnginer has problems?

Could you please grant us support access to your device @Stephen ? Thanks!

@Stephen looks like your balenaEngine has some issues! Let me research better on this next Monday!

Meanwhile you can read more here → How to kill and restart an unhealthy supervsior