Error reading supervisor logs from container

Hello,

Hope everything is running smoothly with you all!

We’ve been Balena’s customers for long and care for this technology.

Today one device from one of our fleets stoped being able to, from the container, get the supervisor’s journal log.
It has been working fine until now and we can’t pin point what might have caused this…

From the host supervisor we can obtain and follow the journalctl, using

journaltcl -f

Making the request from the container now we only receive the response “-- No entries --” after the POST request (as suggested in Balena’s documentation)

curl -X POST -H “Content-Type: application/json” --data ‘{“follow”:false,“all”:true}’ “$BALENA_SUPERVISOR_ADDRESS/v2/journal-logs?apikey=$BALENA_SUPERVISOR_API_KEY”

The host is processing the request
Captura de ecrã 2022-12-14, às 17.17.31

If we use

–data ‘{“follow”:true,“all”:true}’

the shell waits indefinitely for something to arrive,

In other devices from other fleets everything is fine, and in this particular device everything else seems to be working.
We already rebooted the device and the container with no avail.

Hope someone can help solving this issue.
if it is necessary to include more information, we are at your disposal.

All the best!

Hello @suporte could you please confirm if your device is online?

What balenaOS and supervisor versions are you using?

Thanks

Hi,

Do you want me to activate support access?
I can do that if you think it would help, but please be careful this is a production unit.

Thank you

Yes! please grant support access and we can check internally @suporte

Share over DM your device ID!

@suporte is this the only device from your fleet with this problem? do you have other devices that perform well using the same type of connectivity?

Having said that, is the device running properly? Could you please confirm that?

Hi,

It is the only device on the fleet with this problem until now.
All other devices are OK and with logging working well.

Besides the this logging aspect the device seems to be performing all tasks correctly.
It’s main task is data acquisition, that part is OK. It does some actuation too which seems to be OK as far as I know.

Any ideas?

Thank you

Hi, thanks for reporting this. We uncovered than an implementation change to the way logs are retrieved in supervisor v14.4.5 failed to account for persistent logging. I’m working on a fix for this right now.

Here is a link for the PR so you can follow the review progress Fix journalctl to work with persistent logs by pipex · Pull Request #2081 · balena-os/balena-supervisor · GitHub

We will also make sure to ping you when the PR is merged

Hi there. We have just deployed a fix for this issue. Could you upgrade your device supervisor to v14.4.8 and confirm this fixes the problem on your side? Thanks again

Hi Pipex,

Thank you for your speed.
For some reason i am not able to apply the change, do you have any idea why?

Thanks

Hello, being unable to apply the change was a bug, we have released a fix in version 15.25.1 of the dashboard. Could you please try to apply the change again?

Hi,

Version applied, seems that the fix is working, for now everything seems to be running smoothly with the remote unit.

Thank you
Best regards

That’s great to hear. Happy to help!