Sending Docker logs to logz.io

Product support
1

Hello,

Before using Balena we sent all our docker logs to logz.io. I want to continue doing this now that we are using Balena. We are using this docker image: Docker | Logz.io Docs

The problem is that in Balena’s docker-compose file I can’t bind mount, I tried the following:

 logs:
    image: logzio/docker-collector-logs
    labels:
      io.balena.features.balena-socket: '1'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/containers:/var/lib/docker/containers
    environment:
      - LOGZIO_TOKEN
      - LOGZIO_REGION=eu
      - LOGZIO_CODEC=json
    restart: always

But I got an error:

Bind mounts are not allowed

How can I achieve this ?

Thanks for your help,

2

Hi @thibaut,
When using labels in this case the balena-socket feature label, you don’t need the volume mount with it. Can you try this without the volumes and let us know if you run into any issue?

Ref: [Solved] Aggregating logs using Logspout - [Error] Bind mounts are not allowed (a similar issue with logspout)

3

Hello,

Thanks for your answer. The container correctly starts, but I don’t get logs.

After investigating it seems that this logzio container uses filebeat to monitor logfiles with the following configuration:

filebeat.inputs:
- type: container
  paths:
  - /var/lib/docker/containers/*/*.log
  fields_under_root: true

I’m not sure that balena engine store logs in files ?

4

Providing a bit of context: The Balena Supervisor talks directly to the balenaEngine socket to retrieve logs from containers rather than storing logs at specific path (this could very easily fill the storage media and reduce life of SD card given the writes). These logs are then processed and sent to our own log retention system.

You can look if `logz.io can subscribe to the logs for a device, for example using the SDK (see here Balena Node.js SDK - Balena Documentation 138), to consume all output from your service containers.

-Nitish

6

It seems they doesn’t offer this possibility. We’ll probably have to create a component to do this by ourself, or use an other logging provider. Thanks for your answer.

7

In case you have an open balena instance running, you could subscribe to the topic device:{device_number}:logs of the redis service.
You can probably build a short script that subscribe to the topic and push to logz.io.

8

We plan to use a paid plan of Balena cloud, do you know if it’s possible to do it here ?

9

No, with the paid version you can not interact with the redis in the back. Bu you can use the SDK to subscribe to the logs and push them in your infrastructure. Similar to what @nitishagar suggested.

1 Like
10

It’s not clear for me if the SDK should be used on the device or on a backend service ?

11

its something you can install on your laptop or the server, not the IOT device.

You need to authenticate yourself using your balena credentials

Otherwise, try the python sdk

# untested code
from balena import Balena
balena = Balena()
credentials = {'username':<your email>, 'password':<your password>}
balena.auth.login(**credentials)

def send_log_to_server(...):
    ....

def send_error_to_server(...):
    ....
balena.subscribe(
    uuid,
   send_log_to_server,
   send_error_to_server,
, 1)