Service Crash Loop

atwoodjw · May 5, 2022, 8:34pm

I have a multi-container setup running Homebridge, Unifi Controller, and Traefik. At one point, this setup worked great. However, that was several balenaOS, supervisor, and (frankly) unifi-docker releases ago. (I haven’t done a great job applying updates independently.)

After some unknown upgrade, the unifi service seems to be stuck in a loop, where the container starts, runs for a few seconds, dies (or is killed), then the cycle repeats.

What’s a reasonable approach for debugging? I’ve checked the log file in the unifi volume, but there isn’t much there.

[Logs]    [5/5/2022, 1:47:35 PM] Installing service 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:35 PM] Installed service 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:35 PM] Starting service 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:37 PM] Started service 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:37 PM] [unifi] [2022-05-05 13:47:37,835] <docker-entrypoint> Starting unifi controller service.
[Logs]    [5/5/2022, 1:47:37 PM] [unifi] [2022-05-05 13:47:37,835] <docker-entrypoint> Starting unifi controller service.
[Logs]    [5/5/2022, 1:47:38 PM] [unifi] [2022-05-05 13:47:38,059] <docker-entrypoint> WARNING: Running UniFi in insecure (root) mode
[Logs]    [5/5/2022, 1:47:39 PM] Killing service 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:39 PM] [unifi] [2022-05-05 13:47:39,507] <docker-entrypoint> Exit signal received, shutting down
[Logs]    [5/5/2022, 1:47:45 PM] Service exited 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:45 PM] Killed service 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:46 PM] Installing service 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:46 PM] Installed service 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:46 PM] Starting service 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:49 PM] Started service 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:49 PM] [unifi] [2022-05-05 13:47:49,452] <docker-entrypoint> Starting unifi controller service.
[Logs]    [5/5/2022, 1:47:49 PM] [unifi] [2022-05-05 13:47:49,452] <docker-entrypoint> Starting unifi controller service.
[Logs]    [5/5/2022, 1:47:49 PM] [unifi] [2022-05-05 13:47:49,670] <docker-entrypoint> WARNING: Running UniFi in insecure (root) mode
[Logs]    [5/5/2022, 1:47:50 PM] Killing service 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:51 PM] [unifi] [2022-05-05 13:47:51,075] <docker-entrypoint> Exit signal received, shutting down
[Logs]    [5/5/2022, 1:47:56 PM] Service exited 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'
[Logs]    [5/5/2022, 1:47:56 PM] Killed service 'unifi sha256:9fbab0af61c54fa87036e2c39afd08151945a74793ad5e27cd1614e725055cb4'

docker-compose.yml.txt (2.7 KB)
traefik-Dockerfile.txt (60 Bytes)
traefik-rules.toml.txt (202 Bytes)

atwoodjw · May 7, 2022, 12:17am

I run into the same issue w/ the other popular Docker image for Unfi Controller (linuxserver/unifi-controller).

  unifi:
    image: lscr.io/linuxserver/unifi-controller:7.1.61
    environment:
      - "PUID=1000"
      - "PGID=1000"
      - "MEM_LIMIT=1024"
      - "MEM_STARTUP=1024"
    hostname: "unifi.mydomain.com"
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.unifi.entrypoints=websecure"
      - "traefik.http.routers.unifi.rule=Host(`unifi.mydomain.com`)"
      - "traefik.http.services.unifi.loadbalancer.server.scheme=https"
      - "traefik.http.services.unifi.loadbalancer.server.port=8443"
    ports:
      - "3478:3478/udp"
      - "6789:6789/tcp"
      - "8080:8080/tcp"
      - "10001:10001/udp"
    volumes:
      - "unifi_config:/config"

volumes:
  unifi_config:

[Logs]    [5/6/2022, 8:02:02 PM] Installing service 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:02 PM] Installed service 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:02 PM] Starting service 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:05 PM] Started service 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [s6-init] ensuring user provided files have correct perms...exited 0.
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [fix-attrs.d] applying ownership & permissions fixes...
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [fix-attrs.d] done.
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [cont-init.d] executing container initialization scripts...
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [cont-init.d] 01-envfile: executing...
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [cont-init.d] 01-envfile: exited 0.
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [cont-init.d] 01-migrations: executing...
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [migrations] started
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [migrations] no migrations found
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [cont-init.d] 01-migrations: exited 0.
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [cont-init.d] 02-tamper-check: executing...
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [cont-init.d] 02-tamper-check: exited 0.
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [cont-init.d] 10-adduser: executing...
[Logs]    [5/6/2022, 8:02:05 PM] [unifi]
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] -------------------------------------
[Logs]    [5/6/2022, 8:02:05 PM] [unifi]           _         ()
[Logs]    [5/6/2022, 8:02:05 PM] [unifi]          | |  ___   _    __
[Logs]    [5/6/2022, 8:02:05 PM] [unifi]          | | / __| | |  /  \
[Logs]    [5/6/2022, 8:02:05 PM] [unifi]          | | \__ \ | | | () |
[Logs]    [5/6/2022, 8:02:05 PM] [unifi]          |_| |___/ |_|  \__/
[Logs]    [5/6/2022, 8:02:05 PM] [unifi]
[Logs]    [5/6/2022, 8:02:05 PM] [unifi]
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] Brought to you by linuxserver.io
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] -------------------------------------
[Logs]    [5/6/2022, 8:02:05 PM] [unifi]
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] To support LSIO projects visit:
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] https://www.linuxserver.io/donate/
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] -------------------------------------
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] GID/UID
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] -------------------------------------
[Logs]    [5/6/2022, 8:02:05 PM] [unifi]
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] User uid:    1000
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] User gid:    1000
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] -------------------------------------
[Logs]    [5/6/2022, 8:02:05 PM] [unifi]
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [cont-init.d] 10-adduser: exited 0.
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] [cont-init.d] 15-install: executing...
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] *** installing unifi packages ***
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] Selecting previously unselected package unifi.
(Reading database ... 8986 files and directories currently installed.)
[Logs]    [5/6/2022, 8:02:05 PM] [unifi] Preparing to unpack /app/unifi.deb ...
[Logs]    [5/6/2022, 8:02:06 PM] [unifi] Unpacking unifi (7.1.61-17860-1) ...
[Logs]    [5/6/2022, 8:02:06 PM] Killing service 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:06 PM] [unifi] [cont-finish.d] executing container finish scripts...
[Logs]    [5/6/2022, 8:02:06 PM] [unifi] [cont-finish.d] done.
[Logs]    [5/6/2022, 8:02:06 PM] [unifi] [s6-finish] waiting for services.
[Logs]    [5/6/2022, 8:02:07 PM] [unifi] [s6-finish] sending all processes the TERM signal.
[Logs]    [5/6/2022, 8:02:10 PM] [unifi] [s6-finish] sending all processes the KILL signal and exiting.
[Logs]    [5/6/2022, 8:02:11 PM] Service exited 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:11 PM] Killed service 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:12 PM] Installing service 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:12 PM] Installed service 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:12 PM] Starting service 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:15 PM] Started service 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [s6-init] ensuring user provided files have correct perms...exited 0.
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [fix-attrs.d] applying ownership & permissions fixes...
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [fix-attrs.d] done.
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [cont-init.d] executing container initialization scripts...
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [cont-init.d] 01-envfile: executing...
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [cont-init.d] 01-envfile: exited 0.
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [cont-init.d] 01-migrations: executing...
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [migrations] started
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [migrations] no migrations found
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [cont-init.d] 01-migrations: exited 0.
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [cont-init.d] 02-tamper-check: executing...
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [cont-init.d] 02-tamper-check: exited 0.
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [cont-init.d] 10-adduser: executing...
[Logs]    [5/6/2022, 8:02:15 PM] [unifi]
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] -------------------------------------
[Logs]    [5/6/2022, 8:02:15 PM] [unifi]           _         ()
[Logs]    [5/6/2022, 8:02:15 PM] [unifi]          | |  ___   _    __
[Logs]    [5/6/2022, 8:02:15 PM] [unifi]          | | / __| | |  /  \
[Logs]    [5/6/2022, 8:02:15 PM] [unifi]          | | \__ \ | | | () |
[Logs]    [5/6/2022, 8:02:15 PM] [unifi]          |_| |___/ |_|  \__/
[Logs]    [5/6/2022, 8:02:15 PM] [unifi]
[Logs]    [5/6/2022, 8:02:15 PM] [unifi]
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] Brought to you by linuxserver.io
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] -------------------------------------
[Logs]    [5/6/2022, 8:02:15 PM] [unifi]
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] To support LSIO projects visit:
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] https://www.linuxserver.io/donate/
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] -------------------------------------
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] GID/UID
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] -------------------------------------
[Logs]    [5/6/2022, 8:02:15 PM] [unifi]
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] User uid:    1000
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] User gid:    1000
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] -------------------------------------
[Logs]    [5/6/2022, 8:02:15 PM] [unifi]
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [cont-init.d] 10-adduser: exited 0.
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] [cont-init.d] 15-install: executing...
[Logs]    [5/6/2022, 8:02:15 PM] [unifi] *** installing unifi packages ***
[Logs]    [5/6/2022, 8:02:16 PM] [unifi] Selecting previously unselected package unifi.
(Reading database ... 8986 files and directories currently installed.)
[Logs]    [5/6/2022, 8:02:16 PM] [unifi] Preparing to unpack /app/unifi.deb ...
[Logs]    [5/6/2022, 8:02:16 PM] Killing service 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:16 PM] [unifi] Unpacking unifi (7.1.61-17860-1) ...
[Logs]    [5/6/2022, 8:02:16 PM] [unifi] [cont-finish.d] executing container finish scripts...
[Logs]    [5/6/2022, 8:02:16 PM] [unifi] [cont-finish.d] done.
[Logs]    [5/6/2022, 8:02:16 PM] [unifi] [s6-finish] waiting for services.
[Logs]    [5/6/2022, 8:02:17 PM] [unifi] [s6-finish] sending all processes the TERM signal.
[Logs]    [5/6/2022, 8:02:20 PM] [unifi] [s6-finish] sending all processes the KILL signal and exiting.
[Logs]    [5/6/2022, 8:02:21 PM] Service exited 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'
[Logs]    [5/6/2022, 8:02:21 PM] Killed service 'unifi sha256:d72d9197dfc1f4df62f6655733925648096432852b5f7d367630614247f94f36'

When I inspect the volume and find the mountpoint, I see it’s empty, which seems suspicious.

When I SSH into balenaOS, stop resin-supervisor (systemctl stop resin-supervisor) to prevent the crash loop, and start a container manually, it seems to run fine. And even writes files to the volume.

balena run \
  -e PUID=1000 \
  -e PGID=1000 \
  -e MEM_LIMIT=1024 \
  -e MEM_STARTUP=1024 \
  -p 3478:3478/udp \
  -p 6789:6789 `#optional` \
  -p 8880:8880 `#optional` \
  -p 10001:10001/udp \
  -v /var/lib/docker/volumes/1_unifi_config/_data:/config \
  local_image_unifi

So maybe the issue is with resin-supervisor starting the container?

cywang117 · May 7, 2022, 12:30am

Hi,

If you suspect that the Supervisor may be related, a sensible debugging action to complete next is to get the Supervisor logs with journalctl -u balena-supervisor -u resin-supervisor -xef -n 200. Additionally, you could check the exit code of the container when it exits with balena inspect $CONTAINER_ID.

Let us know what kind of logs you see!

Thanks,
Christina

atwoodjw · May 7, 2022, 3:00am

Thanks for the reply.

So it’s pretty difficult to inspect the container, since it only runs for about 10s, before dying / being killed. Then the container is removed and and new one starts and the cycle continues.

The Supervisor logs looks like this, with a loop of service install, start, kill, exit, stop.

May 07 02:56:35 b019c40 balena-supervisor[1009353]: [event]   Event: Service install {"service":{"appId":1,"serviceId":3,"serviceName":"unifi","commit":"localrelease","releaseId":1}}
May 07 02:56:35 b019c40 balena-supervisor[1009353]: [event]   Event: Service installed {"service":{"appId":1,"serviceId":3,"serviceName":"unifi","commit":"localrelease","releaseId":1}}
May 07 02:56:35 b019c40 balena-supervisor[1009353]: [event]   Event: Service start {"service":{"appId":1,"serviceId":3,"serviceName":"unifi","commit":"localrelease","releaseId":1}}
May 07 02:56:37 b019c40 balena-supervisor[1009353]: [event]   Event: Service started {"service":{"appId":1,"serviceId":3,"serviceName":"unifi","commit":"localrelease","releaseId":1}}
May 07 02:56:37 b019c40 balena-supervisor[1009353]: [debug]   Spawning journald with: chroot  /mnt/root journalctl -a -S 2022-05-07 02:56:37 -o json CONTAINER_ID_FULL=2bd2d42e0d7df510695c6814f9eb6978fba5854806c66e8f034d0654a728c9f0
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 7efa08cfdfac4bbbcec55304d79b62d7b12bf25112b79bf0c26a0a9209682ce8
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 8119c0c7798bd7bfbf1719e245b02931dc408d227e50df3e46d117afefa52014
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 5175013ee88c30013af47db437f3185c45c1d3f62dd737ba2c3cf904e3cc9584
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 25a0b483a40f110e7602d7aef9c675fa37fc9073927fb0071a5ef524ad2d7af5
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 6c1cdf9032364192dc7601b187a88a1fda6955b10cd50f52e714d4138443bab1
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   Networks do not match!
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]     Current network:
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   {"aliases":["unifi","2bd2d42e0d7d","unifi.atwoodjw.com"]}
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   New network:
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   {"aliases":["unifi"]}
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   Replacing container for service unifi because of config changes:
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]     Network changes detected
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   Networks do not match!
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]     Current network:
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   {"aliases":["unifi","2bd2d42e0d7d","unifi.atwoodjw.com"]}
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   New network:
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   {"aliases":["unifi"]}
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]   Replacing container for service unifi because of config changes:
May 07 02:56:38 b019c40 balena-supervisor[1009353]: [debug]     Network changes detected
May 07 02:56:39 b019c40 balena-supervisor[1009353]: [event]   Event: Service kill {"service":{"appId":1,"serviceId":3,"serviceName":"unifi","commit":"localrelease","releaseId":1}}
May 07 02:56:40 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 5175013ee88c30013af47db437f3185c45c1d3f62dd737ba2c3cf904e3cc9584
May 07 02:56:40 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 25a0b483a40f110e7602d7aef9c675fa37fc9073927fb0071a5ef524ad2d7af5
May 07 02:56:40 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 6c1cdf9032364192dc7601b187a88a1fda6955b10cd50f52e714d4138443bab1
May 07 02:56:40 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 7efa08cfdfac4bbbcec55304d79b62d7b12bf25112b79bf0c26a0a9209682ce8
May 07 02:56:40 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 8119c0c7798bd7bfbf1719e245b02931dc408d227e50df3e46d117afefa52014
May 07 02:56:40 b019c40 balena-supervisor[1009353]: [debug]   Networks do not match!
May 07 02:56:40 b019c40 balena-supervisor[1009353]: [debug]     Current network:
May 07 02:56:40 b019c40 balena-supervisor[1009353]: [debug]   {"aliases":["unifi","2bd2d42e0d7d","unifi.atwoodjw.com"]}
May 07 02:56:40 b019c40 balena-supervisor[1009353]: [debug]   New network:
May 07 02:56:40 b019c40 balena-supervisor[1009353]: [debug]   {"aliases":["unifi"]}
May 07 02:56:40 b019c40 balena-supervisor[1009353]: [debug]   Replacing container for service unifi because of config changes:
May 07 02:56:40 b019c40 balena-supervisor[1009353]: [debug]     Network changes detected
May 07 02:56:41 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 5175013ee88c30013af47db437f3185c45c1d3f62dd737ba2c3cf904e3cc9584
May 07 02:56:41 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 25a0b483a40f110e7602d7aef9c675fa37fc9073927fb0071a5ef524ad2d7af5
May 07 02:56:41 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 6c1cdf9032364192dc7601b187a88a1fda6955b10cd50f52e714d4138443bab1
May 07 02:56:41 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 7efa08cfdfac4bbbcec55304d79b62d7b12bf25112b79bf0c26a0a9209682ce8
May 07 02:56:41 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 8119c0c7798bd7bfbf1719e245b02931dc408d227e50df3e46d117afefa52014
May 07 02:56:41 b019c40 balena-supervisor[1009353]: [debug]   Networks do not match!
May 07 02:56:41 b019c40 balena-supervisor[1009353]: [debug]     Current network:
May 07 02:56:41 b019c40 balena-supervisor[1009353]: [debug]   {"aliases":["unifi","2bd2d42e0d7d","unifi.atwoodjw.com"]}
May 07 02:56:41 b019c40 balena-supervisor[1009353]: [debug]   New network:
May 07 02:56:41 b019c40 balena-supervisor[1009353]: [debug]   {"aliases":["unifi"]}
May 07 02:56:41 b019c40 balena-supervisor[1009353]: [debug]   Replacing container for service unifi because of config changes:
May 07 02:56:41 b019c40 balena-supervisor[1009353]: [debug]     Network changes detected
May 07 02:56:42 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 6c1cdf9032364192dc7601b187a88a1fda6955b10cd50f52e714d4138443bab1
May 07 02:56:42 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 7efa08cfdfac4bbbcec55304d79b62d7b12bf25112b79bf0c26a0a9209682ce8
May 07 02:56:42 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 8119c0c7798bd7bfbf1719e245b02931dc408d227e50df3e46d117afefa52014
May 07 02:56:42 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 5175013ee88c30013af47db437f3185c45c1d3f62dd737ba2c3cf904e3cc9584
May 07 02:56:42 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 25a0b483a40f110e7602d7aef9c675fa37fc9073927fb0071a5ef524ad2d7af5
May 07 02:56:42 b019c40 balena-supervisor[1009353]: [debug]   Networks do not match!
May 07 02:56:42 b019c40 balena-supervisor[1009353]: [debug]     Current network:
May 07 02:56:42 b019c40 balena-supervisor[1009353]: [debug]   {"aliases":["unifi","2bd2d42e0d7d","unifi.atwoodjw.com"]}
May 07 02:56:42 b019c40 balena-supervisor[1009353]: [debug]   New network:
May 07 02:56:42 b019c40 balena-supervisor[1009353]: [debug]   {"aliases":["unifi"]}
May 07 02:56:42 b019c40 balena-supervisor[1009353]: [debug]   Replacing container for service unifi because of config changes:
May 07 02:56:42 b019c40 balena-supervisor[1009353]: [debug]     Network changes detected
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [event]   Event: Service exit {"service":{"appId":1,"serviceId":3,"serviceName":"unifi","commit":"localrelease","releaseId":1}}
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 8119c0c7798bd7bfbf1719e245b02931dc408d227e50df3e46d117afefa52014
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 5175013ee88c30013af47db437f3185c45c1d3f62dd737ba2c3cf904e3cc9584
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 25a0b483a40f110e7602d7aef9c675fa37fc9073927fb0071a5ef524ad2d7af5
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 6c1cdf9032364192dc7601b187a88a1fda6955b10cd50f52e714d4138443bab1
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 7efa08cfdfac4bbbcec55304d79b62d7b12bf25112b79bf0c26a0a9209682ce8
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [debug]   Networks do not match!
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [debug]     Current network:
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [debug]   {"aliases":["unifi","2bd2d42e0d7d","unifi.atwoodjw.com"]}
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [debug]   New network:
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [debug]   {"aliases":["unifi"]}
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [debug]   Replacing container for service unifi because of config changes:
May 07 02:56:43 b019c40 balena-supervisor[1009353]: [debug]     Network changes detected
May 07 02:56:44 b019c40 balena-supervisor[1009353]: [event]   Event: Service stop {"service":{"appId":1,"serviceId":3,"serviceName":"unifi","commit":"localrelease","releaseId":1}}
May 07 02:56:44 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 5175013ee88c30013af47db437f3185c45c1d3f62dd737ba2c3cf904e3cc9584
May 07 02:56:44 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 25a0b483a40f110e7602d7aef9c675fa37fc9073927fb0071a5ef524ad2d7af5
May 07 02:56:44 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 6c1cdf9032364192dc7601b187a88a1fda6955b10cd50f52e714d4138443bab1
May 07 02:56:44 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 7efa08cfdfac4bbbcec55304d79b62d7b12bf25112b79bf0c26a0a9209682ce8
May 07 02:56:44 b019c40 balena-supervisor[1009353]: [debug]   Found unmanaged Volume: 8119c0c7798bd7bfbf1719e245b02931dc408d227e50df3e46d117afefa52014

This looks suspicious.

Replacing container for service unifi because of config changes:
Network changes detected

Is it possible something in the unifi boot process is changing network settings such that the container is being replaced?

atwoodjw · May 8, 2022, 3:38am

OK, found the issue. Removing hostname from the unifi service in docker-compose.yml resolves the issue. But why is this? Ideally, I’d like for container to have a hostname of unifi.mydomain.com.

myarmolinsky · May 9, 2022, 2:20pm

Hello, it’s possible this may be related to the following github issue: Ignore container hostname from the list of aliases when comparing networks · Issue #1925 · balena-os/balena-supervisor · GitHub
To explain the issue: Moby introduced a new feature in release v20.10.0 where if a hostname different to the container name is added to the service configuration, that hostname will be added to the list of network aliases Add alias for hostname if hostname != container name by olljanat · Pull Request #39204 · moby/moby · GitHub.
For the supervisor this means than when comparing current and target state of the network, these will not match, leading to a restart loop with the message

A temporary fix until the aforementioned balena-supervisor github issue is resolved should be to add the hostname as a network alias on the docker compose configuration, i.e.

version: '2.1'
services:
    my-service:
        hostname: balena.local
        build: ./
        networks:
            default:
                aliases:
                    - balena.local

Please let us know if this indeed appears to be the cause of the issue on your end and if the suggested fix resolves it for you.

atwoodjw · May 9, 2022, 6:56pm

This resolves the issue – thanks! I’ll continue to watch Ignore container hostname from the list of aliases when comparing networks · Issue #1925 · balena-os/balena-supervisor · GitHub.

Topic		Replies	Views
Container stuck continuously restarting Product support	5	7065	April 26, 2019
Services are in a constant restart loop! balenaOS support , balenafin	51	4014	January 15, 2021
Multi-container with Unifi Controller Product support support , raspberrypi3 , docker	10	2454	March 13, 2019
Service start loop on Raspberry pi zero Product support docker , raspberrypizero	2	724	March 23, 2020
Service sometimes fails to restart after crash (error 500) Product support docker	1	451	December 21, 2021

Service Crash Loop

Related topics