For those running openbalena who have observed an error message that regularly appears in device supervisor logs:
[error] LogBackend: server responded with status code: 504
The mystery has finally been solved - and I’d like to share the resolution with the community.
First some background information: device logs (including container logs and other supervisor events) are passed from the device’s balena-supervisor
to open-balena-api
via the endpoint /device/v2/:uuid/log-stream
. Upon receiving a log event, balena-supervisor
opens a connection to the log-stream
endpoint, but rather than streaming log events directly to it, aggregates them in a local buffer first. This buffer then flushes to the log-stream
endpoint on the earlier of aggregating 50 log lines or 60 seconds of inactivity.
We recently added a loki server to our openbalena instance to aggregate and monitor server side logs, which prompted us to look into why devices were routinely failing to post logs to the log-stream
endpoint (which in turn posts those logs to loki). Turns out that it has to do with a timeout mismatch between balena-supervisor
and haproxy-ingress
; haproxy-ingress
has a default connection timeout of 50 seconds, so unless your device generates more than 50 log lines in 50 seconds, it the connection will be timed out by the server, resulting in the LogBackend 504 message showing up on the device, and the log lines not being sent to the log-stream
endpoint.
To solve this, we had to change the haproxy-ingress
default timeouts, which in our case the changes were effected via a helm script config, but depending on your configuration might be implemented differently (however the underlying settings changes should remain the same):
config:
timeout-server: 75s
timeout-server-fin: 75s
timeout-client: 75s
timeout-client-fin: 75s
After deploying with this change, all of the 504 errors should disappear (because balena-supervisor
flushes the log after 60 seconds, before the server aborts the connection), and logs are captured by log-stream
.
Hope this is helps!