# Continuous Container Cycling

**URL:** <https://forums.balena.io/t/continuous-container-cycling/365229>\
**Category:** Product support\
**Created:** [October 24, 2022, 7:08pm UTC](https://forums.balena.io/t/continuous-container-cycling/365229 "2022-10-24T19:08:33Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![SMoore](https://avatars.discourse-cdn.com/v4/letter/s/c5a1d2/32.png) [@SMoore](https://forums.balena.io/u/SMoore)\
**Post date:** [October 24, 2022, 7:08pm UTC](https://forums.balena.io/t/continuous-container-cycling/365229/1 "2022-10-24T19:08:33Z")

</div>

We’ve been tracking down random, sporadic NUC hanging for quite awhile.  
There seems to be a memory-related issue. We’ve put tight memory management on our service containers. Finally we installed fluent-bit to help us get more info out of the system.

Now I am seeing in the fluent-bit logs that balena\_healthcheck and balena\_supervisor go through start-shutdown cycles continuously. Is this normal?

Device: Intel NUC  
BalenaOS: 2.98.33  
Supervisor: 14.0.13

Attached, please find recent

1. healthcheck
2. diagnostics
3. fluent-bit log

[traceFluentBitLog.pdf](https://forums.balena.io/uploads/short-url/AaBnJGN71el45eqFZ3sbcjEn78x.pdf) (36.9 KB)  
[23217009bdacc06c967b41cf0ab1c97e\_diagnostics\_2022.10.24\_15.29.59+0000.pdf](https://forums.balena.io/uploads/short-url/t0tlRb7O5se520DVkmk5jR71S96.pdf) (555.2 KB)  
[23217009bdacc06c967b41cf0ab1c97e\_checks\_2022.10.24\_18.15.44+0000.pdf](https://forums.balena.io/uploads/short-url/hY9sYeX0D3mxxodD13lRLOzi7MI.pdf) (16.9 KB)  
:

---

<div class="post-metadata">

**Author:** ![SMoore](https://avatars.discourse-cdn.com/v4/letter/s/c5a1d2/32.png) [@SMoore](https://forums.balena.io/u/SMoore)\
**Post date:** [October 31, 2022, 4:05pm UTC](https://forums.balena.io/t/continuous-container-cycling/365229/2 "2022-10-31T16:05:26Z")

</div>

Update: the start-shutdown cycles for  
balena\_healthcheck = 3 minutes (create, attach, start, die, destroy)  
balena\_supervisor = 5 minutes (exec\_create, exec\_die)

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [November 1, 2022, 11:05am UTC](https://forums.balena.io/t/continuous-container-cycling/365229/3 "2022-11-01T11:05:24Z")

</div>

Hello @SMoore sorry to hear that you have this performance!

Some questions:

- Is this the only device from your fleet with this issue?
- Did you perform anything to the device before getting into this state?

Could you please grant support access to this device and share on DM the ID of your device?

Thanks

---

<div class="post-metadata">

**Author:** ![SMoore](https://avatars.discourse-cdn.com/v4/letter/s/c5a1d2/32.png) [@SMoore](https://forums.balena.io/u/SMoore)\
**Post date:** [November 1, 2022, 11:39pm UTC](https://forums.balena.io/t/continuous-container-cycling/365229/4 "2022-11-01T23:39:42Z")

</div>

Hi, Marc,  
Thanks for getting back to me.

I sent you a DM with access info.  
Sandy

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [November 2, 2022, 9:07am UTC](https://forums.balena.io/t/continuous-container-cycling/365229/5 "2022-11-02T09:07:02Z")

</div>

Thanks! let us check your device!

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [November 2, 2022, 9:19am UTC](https://forums.balena.io/t/continuous-container-cycling/365229/11 "2022-11-02T09:19:01Z")

</div>

Hello Sandy, i see some weird LogBackend errors. I pinged internally the supervisor team to see how we can help you more.  
On the other hand, is the device performing well for you now?

---

<div class="post-metadata">

**Author:** ![rcooke-warwick](https://avatars.discourse-cdn.com/v4/letter/r/ba9def/32.png) [@rcooke-warwick](https://forums.balena.io/u/rcooke-warwick)\
**Post date:** [November 2, 2022, 12:30pm UTC](https://forums.balena.io/t/continuous-container-cycling/365229/16 "2022-11-02T12:30:59Z")

</div>

Hi there, I also took a look at your device, and at the diagnostics logs that you shared. From both the logs today, and the diagnostics, I don’t see any supervisor container restarts. If you were to take measurements again today, I don’t think that we will see all of these start-\>shutdown cycles.

As to why they were happening when these `fluent-bit` logs were initially taken, I’m not sure - something must have been happening on the device at the time. I find the logs a little hard to parse - could you tell us roughly how frequently/the time between these cycles is?

---

<div class="post-metadata">

**Author:** ![SMoore](https://avatars.discourse-cdn.com/v4/letter/s/c5a1d2/32.png) [@SMoore](https://forums.balena.io/u/SMoore)\
**Post date:** [November 2, 2022, 3:06pm UTC](https://forums.balena.io/t/continuous-container-cycling/365229/18 "2022-11-02T15:06:58Z")

</div>

Hi, Marc,  
Thanks for checking into this. At this moment, the device is behaving well. My expectation is that in a week or two I may see a system crash.

I’m curious about the LogBackend errors. Can you tell me more about this?  
Sandy

---

<div class="post-metadata">

**Author:** ![SMoore](https://avatars.discourse-cdn.com/v4/letter/s/c5a1d2/32.png) [@SMoore](https://forums.balena.io/u/SMoore)\
**Post date:** [November 2, 2022, 6:56pm UTC](https://forums.balena.io/t/continuous-container-cycling/365229/19 "2022-11-02T18:56:08Z")

</div>

Hi, Ryan,  
Thanks for updating me. Here’s some info to answer your questions and clarification:

1. These cycles happen continuously. They are still happening. I’ve attached a parsed log from this morning.
2. If you look in this morning’s log I sent you, column L (name), you will see “balena\_supervisor” highlighted in yellow
3. balena\_supervisor container id stays the same across multiple cycles
4. balena-healthcheck container id changes with every cycle.
5. The timing for the cycles has been:  
balena-healthcheck: every 3 min  
balena\_supervisor: every 5 min

If this is part of the normal function for balenaOS, let me know.

Thanks,  
Sandy

[Balena Trouble Ticket - Fluent-bit Data - Google Sheets.pdf](https://forums.balena.io/uploads/short-url/yUm5d9nBbGHmBiGqNonC67kQIlA.pdf) (78.4 KB)
