Balena Current Release Not Installing Completely

I pinned a new “target release” on a couple of identically configured NUC’s. I have successfully performed this action many times over the last 2 years on these NUC’s.

One NUC installed perfectly & quickly. The other NUC did not, taking many hours to sort-of install with problems as detailed below.

Specifically, on the problem NUC’s Balena Dashboard:

  • “current release” shows the old release
  • “target release” shows the new release.
  • services shows the new release (ie “target release”)

Also, two of the services “admin” and “fluent-bit” are stuck in “Installed” and never progress to “Running.”

Host OS:balenaOS 2.107.8+rev1
Supervisor: 14.4.4

On the problem NUC, I’ve tried (with no resolution):
-stopping all the services
-restarting all the services
-rebooting
-updating to the latest supervisor and balena OS
-re-pinning the target release multiple times

Is there some other action I could take?

Attached please find: screen shot of the Dashboard, health check, diagnostics.

Thanks,
Sandy
Balena Release Incomplete 20221223 - Google Docs.pdf (53.5 KB)
Health_check20221223_fierce_hill.pdf (16.9 KB)
diagnostics20221223_fierce_Hill.pdf (615.6 KB)

Hi,

Can you check the balena ps to see if there are any container left over - as I can see a lot of errors relating to port binding in the logs you attached?

Specifically the port 80 bind error: Bind for 0.0.0.0:80 failed: port is already allocated
Looks related to an open issue: Port already in use, because proxy keeps binding to the wrong container IP · Issue #272 · balena-os/balena-engine · GitHub

The above issue provides some of the steps as workaround. Let us know, if the above works and if we can help further debug and answer questions in the process.

Regards,
Nitish

Hi, Nitish,
Thanks for bringing this to my attention. I followed the steps from the previous linked issue and did not see any improvement. Perhaps I’m missing something critical?

Also Dashboard current release has not updated to target release, even though services show the target release.

Attachments are from after following the steps outlined below.

20221227-HealthCheck-23217009bdacc06c967b41cf0ab1c97e_checks_2022.12.27_19.44.54+0000.pdf (17.4 KB)

20221227-Diagnostics-23217009bdacc06c967b41cf0ab1c97e_diagnostics_2022.12.27_19.44.34+0000.pdf (628 KB)

Hello Sandy, Just confirming you followed the more explicit steps in the comment further down in the linked issue, here is a direct link: Port already in use, because proxy keeps binding to the wrong container IP · Issue #272 · balena-os/balena-engine · GitHub ?

Alan,
Here are the steps I followed from “Port already in use, because proxy keeps binding to the wrong container IP - Issue #272

  • ip link delete supervisor0
  • balena rm --force $(balena ps -aq)
  • balena network prune -f
  • systemctl restart balena
    These steps & the results from following these steps are documented in my previous email.

Let me know if I should be doing other steps etc.
Thanks,
Sandy

Hello Sandy, yes it looks like you followed the proper steps. We’ll take a look at the diagnostics you provided to see if we can troubleshoot further.

Hi, could you try to kill the container (e.x charger-ws_397845_553923) and remove the app-specific bridged docker network (ie. named 123456_blah)? Also, is this running on network mode host?

Hi, did you have any updates regarding this?

Jao,
Thanks for checking in. We have not seen a repeat of this activity.
Sandy