Balena Containers leaving Bridge network

Hi @eeb

Are you still experiencing this issue? Let us know if you still need some help :slight_smile:

Hello,

We are still seeing this issue, we have not updated the Supervisor version to latest across our fleet because, even if we know that there is chance that this is solved, there is no real proof by looking at the Supervisor CHANGELOG or the commits made to the project.

If that’s the only recommendation from you guys, I’ll update the thread once we update the Balena Supervisor version.

Regards,

Hi, just a heads-up that this might happen because you are running NetworkManager in a privileged container, see this comment for details: Containers drop off bridge networks unexpectedly · Issue #258 · balena-os/balena-engine · GitHub
Let me know if this helps?

Hi @robertgzr , so your recommendation would be to remove the privileged setting from this container? I would need to double-check if it affects something internal to our software data collection and management…

Thanks!

@eeb sorry for the delayed response. if the workload doesn’t strictly require it, dropping privileged is usually a good idea.

I bring it up specifically because we’ve seen cases where users were running NetworkManager in a privileged container, which allowed it to interfere with the host OS (we configure the host NM to not manage the container network interfaces).

The thing to check would be if your privileged containers are running NM and to make sure you’re only using it’s client to communicate with the host instance over dbus.

Hi @robertgzr

We dropped the privileged piece but still keeps happening… In the end by trying to implement a self recovery I noticed that the balena engine bind mount (using the docker-compose label) on the containers also stops working.

We end up implementing a self-recover watchdog for this.

If anybody encounters something similar they can verify by noticing that by trying to use the docker sdk for python, or probably any other language, any communication over the mounted docker socket will fail.

Regards,