Release stuck on a loop: Running -> Stopping

On HW running some old resin OS, I upgraded to the latest Balena OS, and then proceeded to upgrade my application to the latest release. This is an app that I have working on the same HW and release on a different Balena cloud project.

But on this upgrade, I see a bunch of containers running correctly, one that switched from Running, to Stopping to Running and so on, and 4 containers stuck on Downloaded.

Don’t see any error messages on the container in question (an MQTT broker).

Any idea what this could be and how to fix?

Thanks

Hi @david-archsys,
Could you provide us a dashboard url to that device and grant us support access to it, so that we can take a look? I can initiate a private message for you to pass me the dashboard url if you prefer it.

Could you also clarify what you mean with

Were the devices pinned on a specific release, or were they moved from a different application before the issue occured?

https://dashboard.balena-cloud.com/devices/a2aa66da42098021ff2b0ffeca1466ba/summary

I meant that I pushed (git push) new container images.

Hi @david-archsys ,
We have recently found an issue that seems related to the logs that your device’s supervisor emits. We already have a fix for this, which will be part of the next OS release.

Are you using EXPOSE in the dockerfile of that service, and either expose: or ports:?
This could be causing the issue in the supervisor version that the device is running.
If that’s the case, then could you try to remove the EXPOSE from it and see how the device behaves?
If that’s not the case then it would be great if you could you pass us part of your dockerfile of that container so that we can further investigate this.

Kind regards,
Thodoris

Hi @thgreasi,

It is just a basic MQTT broker, but here is the basic Dockefile:

FROM erlio/docker-vernemq

USER root

ARG DEBUG
ARG MQTT_USERNAME
ARG MQTT_PASSWORD

RUN apt-get update && \
    apt-get install -y expect

ENV DEBUG=$DEBUG
ENV MQTT_USERNAME=$MQTT_USERNAME
ENV MQTT_PASSWORD=$MQTT_PASSWORD 

There are a couple other lines to run a basic sh script, but for simplicity, I am leaving that out.

@david-archsys
Can you also send us the respective part form the docker-compose?