Datadog agent & /proc errors with Balena labels

Sorry for the repost but I’m trying to figure out how to get datadogs agent to discover and send file system and docker stats. Here’s the problem I’m having - from the balena logs.

22.12.20 19:01:14 (-0500)  datadog  [ AGENT ] 2020-12-23 00:01:14 UTC | WARN | (cgroup_detect.go:126 in parseCgroupMountPoints) | No mountPoints were detected, current cgroup root is: /host/sys/fs/cgroup/
22.12.20 19:01:14 (-0500)  datadog  [ AGENT ] 2020-12-23 00:01:14 UTC | WARN | (checkbase.go:100 in Warnf) | Error collecting containers: could not get cgroups: open /host/proc: no such file or directory
22.12.20 19:01:14 (-0500)  datadog  [ AGENT ] 2020-12-23 00:01:14 UTC | ERROR | (runner.go:289 in work) | Error running check docker: could not get cgroups: open /host/proc: no such file or directory
22.12.20 19:01:22 (-0500)  datadog  [ AGENT ] 2020-12-23 00:01:22 UTC | ERROR | (runner.go:289 in work) | Error running check disk: [{"message": "[Errno 2] No such file or directory: '/host/proc/filesystems'", "traceback": "Traceback (most recent call last):\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/base.py\", line 387, in run\n    self.check(copy.deepcopy(self.instances[0]))\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/disk/disk.py\", line 88, in check\n    self.collect_metrics_psutil()\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/disk/disk.py\", line 99, in collect_metrics_psutil\n    for part in psutil.disk_partitions(all=True):\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/__init__.py\", line 1990, in disk_partitions\n    return _psplatform.disk_partitions(all)\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/_pslinux.py\", line 1144, in disk_partitions\n    with open_text(\"%s/filesystems\" % procfs_path) as f:\n  File \"/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/_common.py\", line 596, in open_text\n    return open(fname, \"rt\", **kwargs)\nIOError: [Errno 2] No such file or directory: '/host/proc/filesystems'\n"}]

And the same issue reported from Datadog:

  • Instance #disk:e5[REDACTED]36f[ERROR]: [{“message”: “[Errno 2] No such file or directory: ‘/host/proc/filesystems’”, “traceback”: “Traceback (most recent call last):\n File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/base/checks/base.py”, line 387, in run\n self.check(copy.deepcopy(self.instances[0]))\n File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/disk/disk.py”, line 88, in check\n self.collect_metrics_psutil()\n File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/datadog_checks/disk/disk.py”, line 99, in collect_metrics_psutil\n for part in psutil.disk_partitions(all=True):\n File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/init.py”, line 1990, in disk_partitions\n return _psplatform.disk_partitions(all)\n File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/_pslinux.py”, line 1144, in disk_partitions\n with open_text(”%s/filesystems" % procfs_path) as f:\n File “/opt/datadog-agent/embedded/lib/python2.7/site-packages/psutil/_common.py”, line 596, in open_text\n return open(fname, “rt”, **kwargs)\nIOError: [Errno 2] No such file or directory: ‘/host/proc/filesystems’\n"}]

  • Instance #docker[ERROR]: could not get cgroups: open /host/proc: no such file or directory

Here’s my docker-compose.yml. Datadogs docs call for mounting some volumes in a way that is not permitted in balena’s flavor of compose files.
My assumption was that the labels io.balena.features.procfs: '1' and io.balena.features.sysfs: '1' would have solved that problem, but perhaps not.

version: '2.1'
    services:
      basicstation:
        build:
          context: ./basicstation
        privileged: true
      datadog:
        image: 'achntrl/datadog-agent:latest'
        privileged: true
        restart: always
        pid: host
        network_mode: host
        labels:
          io.resin.features.dbus: '1'
          io.resin.features.balena-socket: '1'
          io.balena.features.procfs: '1'
          io.balena.features.sysfs: '1'
          io.balena.features.journal-logs: '1'
      losant:
        image: 'losant/edge-agent:latest-arm'
        ports:
          - '8080:8080'

Any suggestions?

Hello,
I haven’t used that, but looking at the docs and the code, the io.balena.features.procfs and io.balena.features.sysfs labels will mount the host’s /proc and /sys folders in /proc and /sys in the container, not /host/proc and /host/sys.

@zvin quick followup question. Are the labels irrelevant when running in privileged:true? Meaning does privileged:true mean essentially the same as defining all the labels.

Hi

That’s right - you either set privileged: true, or use the cap_add and devices field to add permissions one by one in your docker compose. Read more about it here

1 Like