supervisor-log: no such file or directory at boot

Product support
,
1

Hello,

Since my last migration to the latest version of my custom OS based on balenaOS 2.101.11 and supervisor version 14.0.14; one my container is not starting.
This happen only after a “full powercycle reboot” eg. it does NOT happen when :

  • restart services
  • manually start container 30 seconds after boot

Following the “debugging masterclass”, I finally found what seems to be the root cause of my error in journalctl:

level=error msg=“failed to start container”  error=“OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting \“/var/log/supervisor-log\” to rootfs at \“/var/log\” caused: stat /var/log/supervisor-log: no such file or directory: unknown”

Any hint on how to investigate further?
Thx :slight_smile:

2

Hi @luchko ,

Thanks for providing context and error logs. The error message you linked is the typical format for an Engine error message. You can tell because it points to a line the the code of a Go file (container_linux.go:370), and the Engine is the only balenaOS component to my knowledge that’s written in Go (same as Docker Engine). Dissecting the error message further, starting container process tells us that there was an error starting the Supervisor container, with an error of mounting \“/var/log/supervisor-log\” to rootfs at \“/var/log\” caused: stat /var/log/supervisor-log: no such file or directory. This tells us that something is missing in the host OS where it should be present. Since this OS build is a custom one, I would look at creating the /var/log/supervisor-log directory on host if it doesn’t exist.

For more context, the Supervisor container is created with a number of bind mounts using the long syntax, see: meta-balena/start-balena-supervisor at a2287559a0625ca82d362981105353eddcb627ec · balena-os/meta-balena · GitHub

With long syntax, if the path on host is not present, the container start operation will error out instead of proceeding. You can see that bind mounting /var/log/supervisor-log to /var/log is a config specified to the Supervisor container on start.

Let us know if the error persists even if the directory on host exists.

Thanks,
Christina

4

I’m getting this same error now as well. I deployed an image onto my FIN, the containers all launched like they should, then when I put it in local mode for development, I get this error, and none of the containers starting:

Dec 16 02:38:12 bb46e4d balenad[2014]: time="2022-12-16T02:38:12.159547935Z" level=error msg="failed to start container" container=26022092eacb3609772a8d9dc03bef2c18d565b41a1a9e005b41995667114542 error="OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting \"/var/log/supervisor-log\" to rootfs at \"/var/log\" caused: stat /var/log/supervisor-log: no such file or directory: unknown"

SSH’ing into the fin shows that /var/log/supervisor-log folder does exist.

Any ideas how to fix?

5

looks like doing a push to the device fixed it for me

6

Hi @ipearx, could you tell us your OS and supervisor version for the fin? Have you seen this issue occur again?

9

Hello, just a quick message to let you know that I will work on balenaOS again this week, stay yuned :slight_smile: I hope proposed workarround will … work!

10

Thank you! Let us know how it goes!

14

Hello!

Sorry for the long time since my last visit… I was busy upgrading debian version of our whole services.
That was not useless : the service that was not starting now properly works.

I think that could be linked to the AUFS deprecation.
NB: I was using debian-stretch and upgraded to ubuntu focal

Thank you all !