I am using this API to grab journal logs from a container and save to a local storage.
The problem is that it doesn’t send the booting log of the system.
Is there any way to grab the booting log of the system inside a container?
Cheers!
Hi @scarlyon,
If I understand correctly; you can take a look at enabling persistent logging here: Device Logs - Balena Documentation
Thanks for your reply.
However, how can we read these log files from a container?
Cheers.
If you are looking for logs via supervisor API only, then you can try the following: Interacting with the balena Supervisor - Balena Documentation
$> curl "$BALENA_SUPERVISOR_ADDRESS/v2/local/logs"
Let us know if we can help with any other info.
Thanks for the reply.
Does this API return the boot log?
We need to grab the boot & kernel log of the device.
Cheers,
Shane.
No, unfortunately this endpoint does not return the boot logs. There is currently no native way of pulling these logs from within your container. Can I ask what you are trying to accomplish? Perhaps there is another way we can help.
Some of our devices become offline without any reason.
We just want to read the kernel and boot log so that we can figure out what did cause the issue.
Cheers,
Shane.
If you are running this in a lab environment, you can have another device connected to the serial of the device you want boot logs from. So that when it’s booting up you can pull those in on the other device. If this is not in a lab environment, perhaps you can connect to the hostOS and run dmesg to see the logs - if dmesg logs are good enough for you.
If you think one of the application containers is at fault - and not the OS - you can try to enable persistent logging so that the container logs are saved even after your device reboots
Thanks for your tip.
However, the problem is that the device is installed at a production site and there is no way to connect a serial or even display.
We have been using remote SSH to provide access to the cotainers.
Cheers,
Shane.
@anujdeshpande the scenario here is that we have some devices deployed on customer sites with production images.
We run a management container on top of the balenaOS and then run a docker in docker app inside of this container.
We have SSH access into this management container but are locked into i.e. we can get to the OS or balena containers.
For one customer their device is randomly crashing and it seems that it is at the OS level. We have a shared volume mounted to all of our containers where we can write persistent logs to. We are already using the supervisor API to get the docker logs but these dont show any issues, other than docker seems to just randomly stop.
To troubleshoot further we need to be able to see the kernal logs but longer term it would be good to have both the kernal and boot logs written to our shared volume so that we can troubleshoot production issues more easily.
If it is not possible through the API, maybe there is something we do with the config when deploying the balena app to the device to write logs to a sepcific directory?
Hi @scarlyon @g-corrigan
Thanks for explaining your use case. Supervisor dev here – while we have no plans to expose boot logs through the Supervisor API at this time, there is a spec in the works to stream a greater set of logs, even for system services & kernel, to the balena API, where they’ll become available by querying the API. I can’t promise a timeline for this spec but rest assured that we are aware of the use case and the current friction for both users and supports agents due to a lack of ability to access all the device logs easily.
In the meantime, as you mentioned you have access to a Docker client in your container, you can try setting this up: GitHub - balena-io-playground/os-shim: A containerized proof of concept to allow a Docker client to interact with arbitrary files on the host OS without access to bind mounts
The basic premise of that repository is that, because you have access to the host Engine socket, you can tell Docker to spin up a container with the proper bind mounts in order to interface with what you need, in this case journalctl or dmesg directly. A disclaimer, that you should use the concepts in this repo at your own risk, as modifying something on the host OS may lead to devices getting bricked. It has also not been production tested so you’ll have to modify the code to suit your purposes.
Let me know how this works for you!
Regards,
Christina
1 Like
Support agents previously did not mention this method because it is quite hacky (the long-term solution / spec I mentioned above is a better solution), but your use case here is quite restrictive so it’s worth a shot.
Ok, will give it a try.
Thanks!