Loki logging deploy to balena, error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock

Product support
1

I setup logging locally using docker-compose and Grafana/Loki/Promtail, which worked perfectly.

I then went to deploy to balena and it does not function, even after some adjustments such as adding labels.

Promtail container errors for some reason

 promtail  level=error ts=2024-01-19T10:35:33.503019414Z caller=refresh.go:99 component=docker_discovery discovery=docker config=containerlogs/unix:///var/run/docker.sock:80 msg="Unable to refresh target groups" err="error while listing containers: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

Here is the promtail.yml config

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: containerlogs
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: label
            values: ["logging=promtail"]
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'logstream'
      - source_labels: ['__meta_docker_container_label_logging_jobname']
        target_label: 'job'

Here is the relevant part of the compose yml

  promtail:
    image:  grafana/promtail:latest
    container_name: promtail
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: [ "/bin/sh", "-c" ]
    entrypoint:
      - sh
      - -euc
      - |
        cat <<EOF > /etc/promtail/config.yml
        server:
          http_listen_port: 9080
          grpc_listen_port: 0

        positions:
          filename: /tmp/positions.yaml

        clients:
          - url: http://loki:3100/loki/api/v1/push

        scrape_configs:
          - job_name: containerlogs
            docker_sd_configs:
              - host: unix:///var/run/docker.sock
                refresh_interval: 5s
                filters:
                  - name: label
                    values: ["logging=promtail"]
            relabel_configs:
              - source_labels: ['__meta_docker_container_name']
                regex: '/(.*)'
                target_label: 'container'
              - source_labels: ['__meta_docker_container_log_stream']
                target_label: 'logstream'
              - source_labels: ['__meta_docker_container_label_logging_jobname']
                target_label: 'job'
        EOF
        /usr/bin/promtail -config.file /etc/promtail/config.yml
    depends_on:
      - loki
    labels:
        io.balena.features.balena-socket: '1'
        io.balena.features.kernel-modules: '1'
        io.balena.features.journal-logs: '1'
        io.balena.features.sysfs: '1'
        io.balena.features.balena-api: '1'
        io.balena.features.supervisor-api: '1'
    networks:
      - app

  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - logging-config-loki:/etc/loki/
    labels:
        io.balena.features.balena-socket: '1'
        io.balena.features.kernel-modules: '1'
        io.balena.features.journal-logs: '1'
        io.balena.features.sysfs: '1'
        io.balena.features.balena-api: '1'
        io.balena.features.supervisor-api: '1'
    networks:
      - app

  grafana:
    image: grafana/grafana:latest
    ports:
      - 3000:3000
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_DISABLE_LOGIN_FORM=true
    labels:
        io.balena.features.balena-socket: '1'
        io.balena.features.kernel-modules: '1'
        io.balena.features.journal-logs: '1'
        io.balena.features.sysfs: '1'
        io.balena.features.balena-api: '1'
        io.balena.features.supervisor-api: '1'
    networks:
      - app

Am I doing something wrong?