Datadog logs not sent after pushing new release

Product support
1

We use Datadog for log management so alongside our app, we run the Datadog agent.

We’ve noticed that sometimes when we deploy a new release, logs stop getting sent to Datadog. Restarting both services always fixes the issue. This issue doesn’t always happen, and there’s no clear pattern to when it occurs.

Following Datadog’s troubleshooting guide, we SSH’ed into the Datadog container and ran agent status. The following error was reported under container_collect_all:

Status: Error: Error response from daemon: configured logging driver does not support reading

After restarting the services, agent status reports Status: OK (where we previously saw the error).

Here is a simplified version of our set up that exhibits the problem: GitHub - hblumberg/balena-datadog: A buggy attempt to integrate Balena and Datadog.

Has anyone run into a similar issue or know why we might be seeing this particular error?

Thanks in advance!

–

Additional details:

  • Balena Fleet: aarch64 (architecture), Microservices
  • Devices: Raspberry Pi 4 (using 64bit OS), balenaOS 2.94.4
  • Datadog Agent: Docker image datadog/agent-arm64:7.25.0
1 Like
3

Hi @hannah-rt and welcome to the community!

I have not experienced that behavior when I have used Datadog, but admittedly I have not done many builds / pushes to the fleets after setting them up. I know we have a number of Datadog users in the community, so let’s see if any of them chime in over the course of the next few days with any ideas. :slight_smile:

1 Like