I’m trying to track how the balenaos devices are communicating with openbalena, so i tried to do:
docker logs -f openbalena_haproxy_1
This however only seems to log “NOTICE” and “WARNING” messages about the haproxy service state, but i don’t see anything about requests, connections and disconnections. I’ve tried looking at /usr/local/etc/haproxy/haproxy.cfg, but i don’t see any options for logging, or logging levels specifically.
Is it possible to configure the openbalena-haproxy service to log all requests and detailed information about them?
Yes this should be possible. You can change the haproxy configuration according to this documentation: https://www.haproxy.com/documentation/hapee/1-8r1/onepage/#8
You should change the configuration in this file src/haproxy/haproxy.cfg and rebuild the service, but keep in mind, that this file might conflict when you update to a new open balena version, so you might have to redo these changes.
Is rebuilding necessary? I will probably have to test out a few configurations until i get it correct, rebuilding the whole balena service after every change could take a while. Could i just edit the config file inside the docker container and restart that particular service?
You can also edit the file directly in the container, but it won’t persist container recreations. If you just want to experiment with that, changing it in the container and restarting the container should also work just fine.
Well, i was not able to configure the haproxy service to log the requests or anything else for that matter. I tried following this tutorial and a few others, but i seem to be missing some key concepts for how this works.
If someone was able to make the haproxy log requests, i would appreciate if you could share your configs.
# Loads the imudp into rsyslog address space
# and activates it.
# IMUDP provides the ability to receive syslog
# messages via UDP.
$ModLoad imudp
# Address to listen for syslog messages to be
# received.
$UDPServerAddress 0.0.0.0
# Port to listen for the messages
$UDPServerRun 514
# Take the messages of any priority sent to the
# local0 facility (which we reference in the haproxy
# configuration) and send to the haproxy.log
# file.
local0.* -/var/log/haproxy.log
# Discard the rest
& ~
mv rsyslog.conf /etc/rsyslog.d/
Edited the haproxy service /usr/local/etc/haproxy/haproxy.cfg global and default parts to:
global
tune.ssl.default-dh-param 1024
log 0.0.0.0 local0 debug
defaults
mode http
log global
option tcplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
Edited the haproxy service docker-entrypoint.sh to:
#!/bin/sh
set -e
set -o errexit
set -o nounset
readonly RSYSLOG_PID="/var/run/rsyslogd.pid"
# make sure we have rsyslogd's pid file not
# created before
start_rsyslogd() {
rm -f $RSYSLOG_PID
rsyslogd -n 2>&1
}
# first arg is `-f` or `--some-option`
if [ "${1#-}" != "$1" ]; then
set -- haproxy "$@"
fi
if [ "$1" = 'haproxy' ]; then
shift # "haproxy"
# if the user wants "haproxy", let's add a couple useful flags
# -W -- "master-worker mode" (similar to the old "haproxy-systemd-wrapper"; allows for reload via "SIGUSR2")
# -db -- disables background mode
set -- haproxy -W -db "$@"
fi
start_rsyslogd &
exec "$@"
This is somewhat hackish and i’m not completely sure about all these parts (for example installing bash), but seems to be working for me.
Hi,
Thanks for sharing. Glad you got the logs you needed.
You added rsyslog to the haproxy container. I’m not sure it’s really needed for the task you had originally.
Adjusting the log level in haproxy configuration should theoretically be enough to see the request/connection details in the container output.
I tried adjusting the log levels (many permutations), but it only every showed the status of the services. From what i understand haproxy needs some sort of logging service to log it’s output. Does the default balena haproxy service use something like syslog? I tried searching for anything of that sort, but haproxy was the only service running and i did not see any log files.
Before the changes, when i did docker logs -f my-balena-haproxy i would see:
Watches established.
[NOTICE] 317/063640 (16) : New worker #1 (18) forked
[WARNING] 317/063640 (18) : Server backend_api/balena_api_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 317/063640 (18) : backend 'backend_api' has no server available!
[WARNING] 317/063641 (18) : Server backend_registry/balena_registry_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] 317/063641 (18) : backend 'backend_registry' has no server available!
[WARNING] 317/063645 (18) : Server backend_registry/balena_registry_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[WARNING] 317/063650 (18) : Server backend_api/balena_api_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
So i did not see any requests being logged. I tried changing the configuration of haproxy.cfg, but that did not do anything. After doing the config with rsyslog it started showing the request logs.
I tried looking for simple examples where i would not need to use rsyslog, but did not find any, so this was the only way i achieved that. This is the first time i’m configuring haproxy, so i might have missed something. If someone can point out how exactly i should enable request logging without the rsyslog, i would be grateful.
Thanks, this seems to work! I feel a bit foolish, but i guess the default and global fields tripped me up a bit, i thought the logging had to be enabled there.
