Were diagnosing some other things, but the logs are absolutely filled with the following two lines swamping out everything else
Jul 17 02:17:03 acea848 balenad[839]: time=“2019-07-17T02:17:03.942796654Z” level=warning msg=“unknown container” container=af07ce369eb21dfae7f7516c884bf83f90a5ad>
Does anyone know an open issue or something wrong with our config we can fix?
I did find existing:
opened 02:36PM - 16 Mar 19 UTC
closed 06:05PM - 18 Mar 19 UTC
It's not completely clear whether this is an issue with `balena-engine` or elsew… here, but I am creating this issue here so we can point to it while gathering more information about the problem.
The balena-engine logs were showing `"unknown container"` warning messages every 5 minutes on a device:
```
$ journalctl -au balena
...
Mar 16 13:24:10 5e4e1b8 balenad[768]: time="2019-03-16T13:24:10.630748914Z" level=warning msg="unknown container" container=3130da051ce684c7eb549890c8ccd064f7dfa48af4e3ded4e3359f0ed1c5af3c module=libcontainerd names>
Mar 16 13:24:10 5e4e1b8 balenad[768]: Supervisor API: GET /v1/healthy 200 - 15.372 ms
Mar 16 13:24:10 5e4e1b8 balenad[768]: time="2019-03-16T13:24:10.812617790Z" level=warning msg="unknown container" container=3130da051ce684c7eb549890c8ccd064f7dfa48af4e3ded4e3359f0ed1c5af3c module=libcontainerd names>
...
Mar 16 13:29:11 5e4e1b8 balenad[768]: time="2019-03-16T13:29:11.108851935Z" level=warning msg="unknown container" container=3130da051ce684c7eb549890c8ccd064f7dfa48af4e3ded4e3359f0ed1c5af3c module=libcontainerd names>
Mar 16 13:29:11 5e4e1b8 balenad[768]: Supervisor API: GET /v1/healthy 200 - 9.237 ms
Mar 16 13:29:11 5e4e1b8 balenad[768]: time="2019-03-16T13:29:11.293472879Z" level=warning msg="unknown container" container=3130da051ce684c7eb549890c8ccd064f7dfa48af4e3ded4e3359f0ed1c5af3c module=libcontainerd names>
...
Mar 16 13:34:11 5e4e1b8 balenad[768]: time="2019-03-16T13:34:11.590733221Z" level=warning msg="unknown container" container=3130da051ce684c7eb549890c8ccd064f7dfa48af4e3ded4e3359f0ed1c5af3c module=libcontainerd names>
Mar 16 13:34:11 5e4e1b8 balenad[768]: Supervisor API: GET /v1/healthy 200 - 8.888 ms
Mar 16 13:34:11 5e4e1b8 balenad[768]: time="2019-03-16T13:34:11.760298703Z" level=warning msg="unknown container" container=3130da051ce684c7eb549890c8ccd064f7dfa48af4e3ded4e3359f0ed1c5af3c module=libcontainerd names>
...
Mar 16 13:39:12 5e4e1b8 balenad[768]: time="2019-03-16T13:39:12.066166222Z" level=warning msg="unknown container" container=3130da051ce684c7eb549890c8ccd064f7dfa48af4e3ded4e3359f0ed1c5af3c module=libcontainerd names>
Mar 16 13:39:12 5e4e1b8 balenad[768]: Supervisor API: GET /v1/healthy 200 - 15.712 ms
Mar 16 13:39:12 5e4e1b8 balenad[768]: time="2019-03-16T13:39:12.231770847Z" level=warning msg="unknown container" container=3130da051ce684c7eb549890c8ccd064f7dfa48af4e3ded4e3359f0ed1c5af3c module=libcontainerd names>
```
In the logs above, container hash `3130da...` was the balena supervisor container.
The messages _temporarily_ stopped by restarting the supervisor on a new container with the following commands:
```
systemctl stop resin-supervisor
balena rm 3130da051ce6 # ID of the "unknown container"
systemctl start resin-supervisor
```
Within an hour, though, the "unknown container" messages resumed for the new container hash, `d8c06c...`:
```
$ journalctl -au balena
Mar 16 14:49:28 5e4e1b8 balenad[768]: time="2019-03-16T14:49:28.077345776Z" level=warning msg="unknown container" container=d8c06c914643b07b48aac644a9f9f355c71a2363d94062ade512bc9fc9ff82a6 module=libcontainerd names>
Mar 16 14:49:28 5e4e1b8 balenad[768]: Supervisor API: GET /v1/healthy 200 - 12.151 ms
Mar 16 14:49:28 5e4e1b8 balenad[768]: time="2019-03-16T14:49:28.258022018Z" level=warning msg="unknown container" container=d8c06c914643b07b48aac644a9f9f355c71a2363d94062ade512bc9fc9ff82a6 module=libcontainerd names>
...
Mar 16 14:54:28 5e4e1b8 balenad[768]: time="2019-03-16T14:54:28.564795391Z" level=warning msg="unknown container" container=d8c06c914643b07b48aac644a9f9f355c71a2363d94062ade512bc9fc9ff82a6 module=libcontainerd names>
Mar 16 14:54:28 5e4e1b8 balenad[768]: Supervisor API: GET /v1/healthy 200 - 23.670 ms
Mar 16 14:54:28 5e4e1b8 balenad[768]: time="2019-03-16T14:54:28.736068554Z" level=warning msg="unknown container" container=d8c06c914643b07b48aac644a9f9f355c71a2363d94062ade512bc9fc9ff82a6 module=libcontainerd names>
```
**Related moby issue**
A moby issue that also refers to "unknown container" is: https://github.com/moby/moby/issues/35891
<img src="https://frontapp.com/assets/img/icons/favicon-32x32.png" height="16" width="16" alt="Front logo" /> [Front conversations](https://app.frontapp.com/open/top_i1sz)
Hi @jacobrosenthal this was a bug in balena-engine which was fixed in v17.13.4 as well as all the 18.09 versions. Any of the balenaOS versions above 2.32 won’t have this issue anymore. Unfortunately as far as i know it can only be fixed by patching the engine, so upgrading is the best way to reduce the logging.
NP. Happy to upgrade. Thanks for the version hint.
No problemo, happy to help!