I have a multicontainer app running on multiple devices, the app is correctly working on every device with one exception.
On a single device the pod has disappear and I found this error on the supervisor pod:
(node:1) UnhandledPromiseRejectionWarning: Error: (HTTP code 500) server error - layer does not exist
Jul 21 10:59:28 23693f8 d93313190b7a[866]: at /usr/src/app/dist/app.js:596:111352
Jul 21 10:59:28 23693f8 d93313190b7a[866]: at /usr/src/app/dist/app.js:596:111315
Jul 21 10:59:28 23693f8 d93313190b7a[866]: at m.buildPayload (/usr/src/app/dist/app.js:596:111325)
Jul 21 10:59:28 23693f8 d93313190b7a[866]: at IncomingMessage.<anonymous> (/usr/src/app/dist/app.js:596:110825)
Jul 21 10:59:28 23693f8 d93313190b7a[866]: at IncomingMessage.emit (events.js:194:15)
Jul 21 10:59:28 23693f8 d93313190b7a[866]: at endReadableNT (_stream_readable.js:1125:12)
Jul 21 10:59:28 23693f8 d93313190b7a[866]: at process._tickCallback (internal/process/next_tick.js:63:19)
Jul 21 10:59:28 23693f8 d93313190b7a[866]: (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise whic>
Jul 21 10:59:28 23693f8 d93313190b7a[866]: (node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit
I tried to change app on the device through the balena cli, but the supervisor doesn’t seems to respond.
I like to find a way to solve remotely this problem as this device has already been sent to a client.
Do you have any suggestion?
Hi there, when you say you device isn’t working and the pod has disapeared, do you mean that on your device the application containers aren’t there? or the supervisor container is not there? Thanks
I’ll check with the supervisor team for their suggestion. There’s nothing in the dmesg logs you just posted that suggests fs corruption though at least
When the Supervisor logs error messages which include the format (HTTP code ...) <ERROR_MESSAGE> this means that the Supervisor is communicating an error originating from balena Engine’s HTTP API. In this case, a 500 and layer does not exist seems to indicate that the Supervisor cannot find a particular image layer that it has a reference to, thus this results in the device’s invalid state. Is the device currently looping this error state that you shared with logs above? Did this start to happen after an app update?
Also, if you could enable support access to the affected device for 24+ hours and send a link to it here, that would help make debugging faster. Thanks!
I preloaded an app on this device, after a few days I’ve noticed that it wasn’t sending any messagges, ssh into the device and found out that there were no pods.
Then I tried to move the device to another app hoping it’ll start working again, but nothing changed.
I using openbalena, i don’t think I can enable support access… do you have any other way? If not what would you suggest?
Well, one way you could do it is add a balena device to balena cloud, enable support access to it, then add the public ssh key from that device to your other device.
Apart from that, I think this issue is going to be very hard to debug unless we have access ssh access. You could always try upgrading the Host OS and supervisor though. Just make sure you save your data before you do, just in case.
