Container stuck continuously restarting

Product support
1

Hi

I’m using balenaOS 2.32.0+rev2 development build.
I’ve got a problem where my containers are stuck in a loop and are continuously restarting
[Logs] [2019-4-16 17:27:21] Installed service ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’
[Logs] [2019-4-16 17:27:21] Starting service ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’
[Logs] [2019-4-16 17:27:21] Started service ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’
[Logs] [2019-4-16 17:27:22] Killing service ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’
[Logs] [2019-4-16 17:27:23] Service exited ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’
[Logs] [2019-4-16 17:27:23] Killed service ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’
[Logs] [2019-4-16 17:27:23] Installing service ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’
[Logs] [2019-4-16 17:27:23] Installed service ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’
[Logs] [2019-4-16 17:27:23] Starting service ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’
[Logs] [2019-4-16 17:27:24] Started service ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’
[Logs] [2019-4-16 17:27:25] Killing service ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’
[Logs] [2019-4-16 17:27:25] Service exited ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’
[Logs] [2019-4-16 17:27:25] Killed service ‘nginx sha256:295c7be079025306c4f1d65997fcf7adb411c88f139ad1d34b537164aa060369’

I’ve simplified my docker-compose.yml file down to this (I don’t think my issue is related to not using a balena base image, I’m using an x86 machine)
version: ‘2.1’
networks: {}
volumes:
resin-data: {}
services:
nginx:
image: nginx:1.14.2
ports:
- “80:80”

Looking at the journalctl output for resin-supervisor, shows these messages being logged every few seconds:

Apr 16 16:28:02 25171d6 resin-supervisor[1907]: Replacing container for service nginx because of config changes:
Apr 16 16:28:02 25171d6 resin-supervisor[1907]:   Array Fields:  expose
Apr 16 16:28:02 25171d6 resin-supervisor[1907]: Replacing container for service nginx because of config changes:
Apr 16 16:28:02 25171d6 resin-supervisor[1907]:   Array Fields:  expose
Apr 16 16:28:02 25171d6 resin-supervisor[1907]: Replacing container for service nginx because of config changes:
Apr 16 16:28:02 25171d6 resin-supervisor[1907]:   Array Fields:  expose

I don’t think that this is environment variable related, because if I remove the “ports” block from my docker-compose.yml file, then the container starts up OK and doesn’t get restarted.
If I do this then I’m not sure how to forward ports into the container?

Is this a known issue? Does anyone have any ideas?
Thanks!

2 Likes
2

This happened to us as well

Upgrading to 2.32

An nginx service in docker compose file kept restarting with the error quoted above

Nginx also reported an inability to create/open /var/run/nginx.pid, this was a red herring

I changed to use

expose:
  - '80'

instead of

ports:
  - 80:80

and the infinite restart stopped

can anyone comment on the cause? it is extremely confusing to troubleshoot from the symptom :stuck_out_tongue:

4

This issue was seen on BalenaOS 2.32.0+rev2.prod.

I have now “worked around” this issue by downgrading to BalenaOS 2.31.2.

5

Same story, all our containers with port mappings in the docker-compose config are restarting constantly on 2.32.0+rev2.prod

6

@peterl, @jotham, @thiver, thank you for reporting this issue. I believe this issue is an instance of:

Quoting the issue:

When an EXPOSE instruction is used in a Dockerfile , and an overlapping expose or ports instruction is also used in the docker-compose.yml file, the supervisor will restart the app container in a loop.
The temporary workaround is to remove the overlapping setting from either the Dockerfile or the docker-compose.yml file.

A supervisor code fix has already been merged and is in the process of integration with a new balenaOS release. The whole process (with OS testing for all the different device types) usually takes a couple of weeks.

1 Like
7

Hi @pdcastro, thanks for your response.

All my restarting containers did indeed have an EXPOSE instruction in the Dockerfile itself, while having a ports instruction in the docker-compose.yml file.