Datadog agent & /proc errors with Balena labels

Product support balenaOS
1

Sorry for the repost but I’m trying to figure out how to get datadogs agent to discover and send file system and docker stats. Here’s the problem I’m having - from the balena logs.

22.12.20 19:01:14 (-0500)  datadog  [ AGENT ] 2020-12-23 00:01:14 UTC | WARN | (cgroup_detect.go:126 in parseCgroupMountPoints) | No mountPoints were detected, current cgroup root is: /host/sys/fs/cgroup/
22.12.20 19:01:14 (-0500)  datadog  [ AGENT ] 2020-12-23 00:01:14 UTC | WARN | (checkbase.go:100 in Warnf) | Error collecting containers: could not get cgroups: open /host/proc: no such file or directory
22.12.20 19:01:14 (-0500)  datadog  [ AGENT ] 2020-12-23 00:01:14 UTC | ERROR | (runner.go:289 in work) | Error running check docker: could not get cgroups: open /host/proc: no such file or directory
22.12.20 19:01:22 (-0500)  datadog  [ AGENT ] 2020-12-23 00:01:22 UTC | ERROR | (runner.go:289 in work) | Error running check disk: [{"message": "[Errno 2] No such file or directory: '/host/proc/filesystems'", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/base.py\", line 387, in run\n    self.check(copy.deepcopy(self.instances[0]))\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/disk/disk.py\", line 88, in check\n    self.collect_metrics_psutil()\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/disk/disk.py\", line 99, in collect_metrics_psutil\n    for part in psutil.disk_partitions(all=True):\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/__init__.py\", line 1990, in disk_partitions\n    return _psplatform.disk_partitions(all)\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/_pslinux.py\", line 1144, in disk_partitions\n    with open_text(\"%s/filesystems\" % procfs_path) as f:\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/_common.py\", line 596, in open_text\n    return open(fname, \"rt\", **kwargs)\nIOError: [Errno 2] No such file or directory: '/host/proc/filesystems'\n"}]

And the same issue reported from Datadog:

  • Instance #disk:e5[REDACTED]36f[ERROR]: [{“message”: “[Errno 2] No such file or directory: ‘/host/proc/filesystems’”, “traceback”: “Traceback (most recent call last):\n File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/base.py", line 387, in run\n self.check(copy.deepcopy(self.instances[0]))\n File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/disk/disk.py", line 88, in check\n self.collect_metrics_psutil()\n File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/disk/disk.py", line 99, in collect_metrics_psutil\n for part in psutil.disk_partitions(all=True):\n File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/init.py", line 1990, in disk_partitions\n return _psplatform.disk_partitions(all)\n File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/_pslinux.py", line 1144, in disk_partitions\n with open_text("%s/filesystems" % procfs_path) as f:\n File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/_common.py", line 596, in open_text\n return open(fname, "rt", **kwargs)\nIOError: [Errno 2] No such file or directory: ‘/host/proc/filesystems’\n”}]

  • Instance docker[ERROR]: could not get cgroups: open /host/proc: no such file or directory

Here’s my docker-compose.yml. Datadogs docs call for mounting some volumes in a way that is not permitted in balena’s flavor of compose files.
My assumption was that the labels io.balena.features.procfs: '1' and io.balena.features.sysfs: '1' would have solved that problem, but perhaps not.

version: '2.1'
    services:
      basicstation:
        build:
          context: ./basicstation
        privileged: true
      datadog:
        image: 'achntrl/datadog-agent:latest'
        privileged: true
        restart: always
        pid: host
        network_mode: host
        labels:
          io.resin.features.dbus: '1'
          io.resin.features.balena-socket: '1'
          io.balena.features.procfs: '1'
          io.balena.features.sysfs: '1'
          io.balena.features.journal-logs: '1'
      losant:
        image: 'losant/edge-agent:latest-arm'
        ports:
          - '8080:8080'

Any suggestions?

4

Hello,
I haven’t used that, but looking at the docs and the code, the io.balena.features.procfs and io.balena.features.sysfs labels will mount the host’s /proc and /sys folders in /proc and /sys in the container, not /host/proc and /host/sys.

6

@zvin quick followup question. Are the labels irrelevant when running in privileged:true? Meaning does privileged:true mean essentially the same as defining all the labels.

7

Hi

That’s right - you either set privileged: true, or use the cap_add and devices field to add permissions one by one in your docker compose. Read more about it here

1 Like