Unable to get Mosquitto operating on BalenaOS

Stephen · January 24, 2023, 11:30am

I have a mosquitto container that is relaying messages to another container that is performing some calculations. No matter what configurations are applied the mosquitto container always falls over approximately every 6 minutes.

I have run the exact same configuration on the same hardware using Ubuntu and it runs without a problem, but as soon as BalenaOS is introduced the 6 minute failure appears. There are unfortunately no error messages to assist in the troubleshooting, but it does not seem to be mosquitto configuration related.

Underlying architecture: Amd64
Dockerfile Base Image utilised: balenalib/amd64-alpine:3.16-run

I have tried a few variations for the mosquitto Dockerfile. The latest variation is using the Dockerfile from the official eclipse-mosquitto page and replacing the base image with the balenalib equivalent to ensure both the balena environment and mosquitto requirements are catered for. This still runs well on ubuntu but not BalenaOS.

Official Mosquitto Dockerfile https://github.com/eclipse/mosquitto/blob/a8448a9c7b14bdaee6ec80419d43fd6544e789b6/docker/2.0/Dockerfile

Just for reference I did see that a mosquitto balenablock was created but I would prefer to use a simple Dockerfile for my mosquitto implementation.

mpous · January 24, 2023, 11:45am

Hello @Stephen welcome to the balena community!

Could you please share the docker-compose that you are using? I would like to test! And could you please share the logs when Mosquitto falls over?

Do you have any special configuration? Do you use any version in particular?

Thanks!

Stephen · January 24, 2023, 12:07pm

Hi Mark,

Thanks for your message.

You can utilise this for the docker-compose.yml

version: '2.4'

volumes:
  model:
  mosquitto:

networks:
  local-net:
    driver: bridge

services:

  mqtt-broker:
    build: ./mosquitto
    container_name: mqtt-broker
    networks:
      - local-net  
    restart: always
    ports:
      - 1883:1883
      - 9001:9001
    volumes:
      - mosquitto:/mosquitto/data
      - mosquitto:/mosquitto/log

  # simple_model:
  #   image: simple_model:latest
  #   container_name: simple_model
  #   networks:
  #     - local-net
  #   restart: always
  #   environment:
  #     - BROKER_ADDRESS=mqtt-broker
  #     - BROKER_PORT=1883
  #   volumes:
  #     - 'model:/data'

Here is the dockerfile:

FROM balenalib/amd64-alpine:3.16-run

LABEL maintainer="Roger Light <roger@atchoo.org>" \
    description="Eclipse Mosquitto MQTT Broker"

ENV VERSION=2.0.15 \
    DOWNLOAD_SHA256=4735b1d32e3f91c7a8896741d88a3022e89730a1ee897946decfa0df27039ac6 \
    GPG_KEYS=A0D6EEA1DCAE49A635A3B2F0779B22DFB3E717B7 \
    LWS_VERSION=4.2.1 \
    LWS_SHA256=842da21f73ccba2be59e680de10a8cce7928313048750eb6ad73b6fa50763c51

RUN set -x && \
    apk --no-cache add --virtual build-deps \
        build-base \
        cmake \
        cjson-dev \
        gnupg \
        libressl-dev \
        linux-headers \
        util-linux-dev && \
    wget https://github.com/warmcat/libwebsockets/archive/v${LWS_VERSION}.tar.gz -O /tmp/lws.tar.gz && \
    echo "$LWS_SHA256  /tmp/lws.tar.gz" | sha256sum -c - && \
    mkdir -p /build/lws && \
    tar --strip=1 -xf /tmp/lws.tar.gz -C /build/lws && \
    rm /tmp/lws.tar.gz && \
    cd /build/lws && \
    cmake . \
        -DCMAKE_BUILD_TYPE=MinSizeRel \
        -DCMAKE_INSTALL_PREFIX=/usr \
        -DDISABLE_WERROR=ON \
        -DLWS_IPV6=ON \
        -DLWS_WITHOUT_BUILTIN_GETIFADDRS=ON \
        -DLWS_WITHOUT_CLIENT=ON \
        -DLWS_WITHOUT_EXTENSIONS=ON \
        -DLWS_WITHOUT_TESTAPPS=ON \
        -DLWS_WITH_EXTERNAL_POLL=ON \
        -DLWS_WITH_HTTP2=OFF \
        -DLWS_WITH_SHARED=OFF \
        -DLWS_WITH_ZIP_FOPS=OFF \
        -DLWS_WITH_ZLIB=OFF && \
    make -j "$(nproc)" && \
    rm -rf /root/.cmake && \
    wget https://mosquitto.org/files/source/mosquitto-${VERSION}.tar.gz -O /tmp/mosq.tar.gz && \
    echo "$DOWNLOAD_SHA256  /tmp/mosq.tar.gz" | sha256sum -c - && \
    wget https://mosquitto.org/files/source/mosquitto-${VERSION}.tar.gz.asc -O /tmp/mosq.tar.gz.asc && \
    export GNUPGHOME="$(mktemp -d)" && \
    found=''; \
    for server in \
        hkps://keys.openpgp.org \
        hkp://keyserver.ubuntu.com:80 \
        pgp.mit.edu \
    ; do \
        echo "Fetching GPG key $GPG_KEYS from $server"; \
        gpg --keyserver "$server" --keyserver-options timeout=10 --recv-keys "$GPG_KEYS" && found=yes && break; \
    done; \
    test -z "$found" && echo >&2 "error: failed to fetch GPG key $GPG_KEYS" && exit 1; \
    gpg --batch --verify /tmp/mosq.tar.gz.asc /tmp/mosq.tar.gz && \
    gpgconf --kill all && \
    rm -rf "$GNUPGHOME" /tmp/mosq.tar.gz.asc && \
    mkdir -p /build/mosq && \
    tar --strip=1 -xf /tmp/mosq.tar.gz -C /build/mosq && \
    rm /tmp/mosq.tar.gz && \
    make -C /build/mosq -j "$(nproc)" \
        CFLAGS="-Wall -O2 -I/build/lws/include -I/build" \
        LDFLAGS="-L/build/lws/lib" \
        WITH_ADNS=no \
        WITH_DOCS=no \
        WITH_SHARED_LIBRARIES=yes \
        WITH_SRV=no \
        WITH_STRIP=yes \
        WITH_TLS_PSK=no \
        WITH_WEBSOCKETS=yes \
        prefix=/usr \
        binary && \
    addgroup -S -g 1883 mosquitto 2>/dev/null && \
    adduser -S -u 1883 -D -H -h /var/empty -s /sbin/nologin -G mosquitto -g mosquitto mosquitto 2>/dev/null && \
    mkdir -p /mosquitto/config /mosquitto/data /mosquitto/log && \
    install -d /usr/sbin/ && \
    install -s -m755 /build/mosq/client/mosquitto_pub /usr/bin/mosquitto_pub && \
    install -s -m755 /build/mosq/client/mosquitto_rr /usr/bin/mosquitto_rr && \
    install -s -m755 /build/mosq/client/mosquitto_sub /usr/bin/mosquitto_sub && \
    install -s -m644 /build/mosq/lib/libmosquitto.so.1 /usr/lib/libmosquitto.so.1 && \
    install -s -m755 /build/mosq/src/mosquitto /usr/sbin/mosquitto && \
    install -s -m755 /build/mosq/apps/mosquitto_ctrl/mosquitto_ctrl /usr/bin/mosquitto_ctrl && \
    install -s -m755 /build/mosq/apps/mosquitto_passwd/mosquitto_passwd /usr/bin/mosquitto_passwd && \
    install -s -m755 /build/mosq/plugins/dynamic-security/mosquitto_dynamic_security.so /usr/lib/mosquitto_dynamic_security.so && \
    install -m644 /build/mosq/mosquitto.conf /mosquitto/config/mosquitto.conf && \
    install -Dm644 /build/lws/LICENSE /usr/share/licenses/libwebsockets/LICENSE && \
    install -Dm644 /build/mosq/epl-v20 /usr/share/licenses/mosquitto/epl-v20 && \
    install -Dm644 /build/mosq/edl-v10 /usr/share/licenses/mosquitto/edl-v10 && \
    chown -R mosquitto:mosquitto /mosquitto && \
    apk --no-cache add \
        ca-certificates \
        cjson \
        libressl && \
    apk del build-deps && \
    rm -rf /build

VOLUME ["/mosquitto/data", "/mosquitto/log"]

RUN echo 'listener 1883' >> /mosquitto/config/mosquitto.conf
RUN echo 'allow_anonymous true' >> /mosquitto/config/mosquitto.conf
RUN echo 'persistence true' >> /mosquitto/config/mosquitto.conf
RUN echo 'persistence_location /mosquitto/data/' >> /mosquitto/config/mosquitto.conf
RUN echo 'pid_file /mosquitto/data/mosquitto.pid' >> /mosquitto/config/mosquitto.conf
RUN echo 'user mosquitto' >> /mosquitto/config/mosquitto.conf
RUN echo 'log_dest stdout' >> /mosquitto/config/mosquitto.conf
RUN echo 'log_dest file /mosquitto/log/mosquitto.log' >> /mosquitto/config/mosquitto.conf
RUN echo 'log_type all' >> /mosquitto/config/mosquitto.conf
RUN echo 'connection_messages true' >> /mosquitto/config/mosquitto.conf

RUN chown mosquitto: /mosquitto/config
USER mosquitto

EXPOSE 1883
CMD ["/usr/sbin/mosquitto", "-c", "/mosquitto/config/mosquitto.conf"]

mpous · January 24, 2023, 12:55pm

@Stephen could you please share the logs when the mosquitto restarts?

On the other hand, did you try to build from the docker image from Eclipse mosquitto (e.g. below from balenAir) or do you need something specific?

  mqtt:
    image: eclipse-mosquitto:1.6.15
    ports:
      - "1883:1883"
    restart: always

Stephen · January 25, 2023, 6:15am

Hi @mpous, I actually never tested that, although thinking now I definitely should have.
I ran this for both version 1.6.15 and 2.0.15 with the same result being that they shut down just like before.

The logs are as follows:

mqtt  1674626640: Config loaded from /mosquitto/config/mosquitto.conf.
 mqtt  1674626640: Opening ipv4 listen socket on port 1883.
 mqtt  1674626640: Opening ipv6 listen socket on port 1883.
 mqtt  1674626640: mosquitto version 1.6.15 running
 mqtt  1674626861: mosquitto version 1.6.15 terminating
 mqtt  1674627130: mosquitto version 1.6.15 starting

Stephen · January 25, 2023, 7:40am

I ran the health checks and the check container engine failed.
I have uploaded the device diagnostics check as well.

device_diagnostics.txt (1.3 MB)

Stephen · January 26, 2023, 11:39am

Hi @mpous,

Do you know what causes this issue? Is there something to learn here or must I just reformat the device?

Kind regards,

mpous · January 26, 2023, 12:13pm

Looks like the supervisor or the balenaEnginer has problems?

Could you please grant us support access to your device @Stephen ? Thanks!

mpous · January 27, 2023, 4:11pm

@Stephen looks like your balenaEngine has some issues! Let me research better on this next Monday!

Meanwhile you can read more here → How to kill and restart an unhealthy supervsior