Containers still stuck in a bad state after container deletion

Project help
,
1

Hi,

I am supporting a project, we have +1 raspberrypi3 working fine until a month ago. A single rasp is having a weird behavior.

In short, one of our containers (NTP) stuck in a bad state.
It tells that port 123 is in use.

  • At host, I killed the process that was using the port 123
  • I deleted the container
  • I rebooted the rasp

When it turns on, balena recreates the NTP container but this new container stuck in the same error:

Bind for 0.0.0.0:123 failed: port is already allocated

Any thought about what I can do?

BTW, another service is weird as well…
We are running a Pihole container and in this specific rasp, this service is restating telling us that is missing a binary file:

/opt/pihole/webpage.sh: line 380: source: /etc/pihole/setupVars.conf: cannot execute binary file
[cont-init.d] 10-custom.sh: exited 126.
[cont-finish.d] executing container finish scripts…
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

All rasps are running the same services but this single one is having this misbehavior.

Any help is appreciated <3

Cheers,
tdruiva

2

Hi @tdruiva,

Regarding your first issue with the port is already allocated error message, we have some steps to fix the issue here: Port already in use, because proxy keeps binding to the wrong container IP · Issue #272 · balena-os/balena-engine · GitHub

As there is no clear way to reproduce, it has been tough to find the root cause of this GitHub issue. What version of the OS are you running? Could you try upgrading to the latest OS version? Newer OSes contain bugfixes and improvements that may help.

Regarding your second issue, it’s likely some file is missing or in the wrong format in the Pihole container filesystem. This could be caused by some part of the container build process being interrupted on build, or becoming corrupted through Engine malfunction or non-atomic writes getting interrupted. I’m not sure what the Pihole startup process is so can’t say for sure. If you’re looking to resolve this issue, I’d recommend deleting the container and restarting the Supervisor to trigger a container rebuild, with the following:

systemctl stop balena-supervisor
balena rm $(balena ps -aq -f name=pihole)
systemctl start balena-supervisor

Note that balena-supervisor should be replaced with resin-supervisor in older OSes (2.7x or lower but can’t recall the exact semver). Let us know how it goes!

Thanks,
Christina

4

Hey Christina,

Regarding your first issue with the port is already allCocated error message, we have some steps to fix the issue here: Port already in use, because proxy keeps binding to the wrong container IP · Issue #272 · balena-os/balena-engine · GitHub

This helped me with the port is already alocated.
Thank you for your support <3

As there is no clear way to reproduce, it has been tough to find the root cause of this GitHub issue. What version of the OS are you running? Could you try upgrading to the latest OS version? Newer OSes contain bugfixes and improvements that may help.

This is my current version: “2.94.4”
I tried to update but I guess this is the lattest version.

Regarding your second issue, it’s likely some file is missing or in the wrong format in the Pihole container filesystem. This could be caused by some part of the container build process being interrupted on build, or becoming corrupted through Engine malfunction or non-atomic writes getting interrupted. I’m not sure what the Pihole startup process is so can’t say for sure. If you’re looking to resolve this issue, I’d recommend deleting the container and restarting the Supervisor to trigger a container rebuild, with the following:

systemctl stop balena-supervisor
balena rm $(balena ps -aq -f name=pihole)
systemctl start balena-supervisor

Note that balena-supervisor should be replaced with resin-supervisor in older OSes (2.7x or lower but can’t recall the exact semver). Let us know how it goes!
I tried those steps above, I also deleted the image this pihole container was
using and when it got recreated, the issue still there :confused:
Weirrrdd…

Well, I just changed /etc/pihole/setupVars.conf content and everything
worked.

Thank you so much for being with me <3

tdruiva