zram: Decompression failed! err=-22

I’ve had some issues recently with my devices failing with a “zram: Decompression failed!” error in the dmesg logs.

Sometimes the unit runs for a while before this issue occurs, sometimes it can happen quite quickly. I’m running 4 containers on a BalenaFin v1.1.1, including InfluxDB and Telegraf to collect system metrics to check on the health of my devices (ironically to monitor the memory usage of my device to catch/prevent OOM errors with InfluxDB). The other 2 containers are a GUI and a python based data uploader (small packets of data in the 10’s of kB), which don’t consume much memory (10-20MB each reported by Balena)

As far as I can tell, I’m not maxing out the memory, unless something big is happening between logs (every 15seconds).

Setup:

  • BalenaFin v1.1.1
  • 4 Containers + Supervisor (And occasional health-check container the supervisor runs)
  • OS version: balenaOS 2.80.3+rev1
  • Supervisor version: 12.7.0
  • Typical memory used around 60-65% when issue occurs
  • Peak memory used has been as high as 75%

Below is a screen grab of the system metrics with a data gap in the middle (where the white markers are) where the unit stopped working, and then restarted a little later (although in this instance it didn’t seem to start 2 of the containers properly for some reason & I had to restart them manually)

Anyone run into this before?

I can provide some more dmesg output if needed.

Hi, there is a reported kernel problem that feels similar fixed in v5.10 (see kernel/git/torvalds/linux.git - Linux kernel source tree).
We are working on supporting the 5.10 kernel for our RaspberryPi family of devices including the balenaFin. I have linked the github issue tracking that work (Community user is requesting kernel 5.10 for RPI · Issue #601 · balena-os/balena-raspberrypi · GitHub) here and we will update here once it’s ready.

Hey @ST-Mono ,

Just to let you know that we’ve just merged support for kernel 5.10.

Cheers!

Great! I look forward to testing it once its available on the BalenaCloud dashboard to upgrade to.

Any idea when there will be a release of BalenaOS with kernel 5.10 for the BalenaFin available via BalenaCloud?

Hey @st-mono I’ve opened an issue with our devices team as we prioritize testing and releasing versions for each device type. You can track the progress here, and once released we will update this thread.

Hey @st-mono, just a heads up that v2.83.21+rev1 has been released to production for Balena Fin with kernel 5.10!

Hi @klutchell, thanks for the update. We are now testing this release and haven’t seen any zram errors so far (just a few other bugs of our own doing to sort out now :slight_smile: ).

Great, glad we seem to have avoided further zram errors. Let us know if there is anything else!