Hello, when i push my project to balenaOs load 100%, but when i go to see the services runing, some services stay in downloaded as the image below.
Hello, @morais !
Thanks for reporting in. We need some more information to better understand what’s going on. Could you please grab a larger screenshot that also includes the device metrics panel (top-right). I’m curious if your device is running out of available memory.
Also it would be very helpful to know what kind of device you’re using and the balenaOS version.
Thanks!
We have been experiencing the same issue (containers staying in downloaded and not starting) and reported it to chat support.
We are running balenaOS 2.69.1+rev1 with supervisor 12.3.5 on a Raspberry Pi 4.
We would really appreciate the support on this.
Thank you 
Hello folks, as my colleague mentioned we would need more context on your specific situation. The containers staying in the “downloaded” state and not starting the containers could be because of a number of factors. Please let us know what you see in the logs, what are the device metrics of the device. Also n42, I do see another thread open on the forums from you and if we can troubleshoot the issue there?
reported it to chat support
If your conversation is ongoing on chat support, then I would recommend to follow-up there only.
Hello, @andrewnhem!
Yes, I can!
but i can make all services run after i use the option purge in the balena cloud.
but, this problem occurs interminably.
Hi @Morais, when this problem manifests, can you please (1) run Device Diagnostics accessed from the device side bar menu and (2) check the supervisor logs by running the command journalctl -fn 1000 -u resin-supervisor on HostOS?
I wonder if you are experiencing this bug:
We already have a work-in-progress solution to this bug.
Even if you are not experiencing this bug, it’d help to see the supervisor logs. Purge Data deletes all named volumes and recreates them as empty. So if purging data resolves the issue for a short time, the application might be pushing the device’s limits.
Is your application writing high volume of data? Could it be that the device runs out of space?
ok, in the next time, i’ll try run this command and chek the logs of supervisor.
about the option purge, yeah, i know this command clean my data and volumes, maybe some volumes is incorrect and this problem is caused by this, but i have another problem like this, in others applications.
when i take this logg, i’ll post here.
Hi, this the log of supervisor and image.
– Logs begin at Fri 2021-03-05 16:49:28 UTC. –
Mar 05 16:49:30 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:49:38 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:49:40 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:49:55 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:49:57 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:50:01 9d819d2 resin-supervisor[3717]: [error] Error from the API: 503
Mar 05 16:50:01 9d819d2 resin-supervisor[3717]: [error] Non-200 response from the API! Status code: 503 - message: Error
Mar 05 16:50:01 9d819d2 resin-supervisor[3717]: [error] at /usr/src/app/dist/app.js:22:554765
Mar 05 16:50:01 9d819d2 resin-supervisor[3717]: [error] at runMicrotasks ()
Mar 05 16:50:01 9d819d2 resin-supervisor[3717]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Mar 05 16:50:01 9d819d2 resin-supervisor[3717]: [error] at async /usr/src/app/dist/app.js:22:554073
Mar 05 16:50:01 9d819d2 resin-supervisor[3717]: [error] at async /usr/src/app/dist/app.js:22:555595
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [info] Applying target state
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName, stdinOpen
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:13 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:14 9d819d2 resin-supervisor[3717]: [event] Event: Image removal {“image”:{“id”:73,“name”:“registry2.balena-cloud.com/v2/839167ee9ca70963e738a43288840ed8@sha256:6a9bcd8337e513ee6738e145bca0a11284003ade110e5a3dfe9fa339aac4adb9",“appId”:1778390,“serviceId”:942848,“serviceName”:“pocfurukawa”,“imageId”:3353061,“releaseId”:1718715,“dependent”:0,“dockerImageId”:"sha256:1a4449ab18120e87e42f744c94f32b698c198840c0e7b2f19f61bcf588828282”}}
Mar 05 16:50:14 9d819d2 resin-supervisor[3717]: [event] Event: Image removed {“image”:{“id”:73,“name”:“registry2.balena-cloud.com/v2/839167ee9ca70963e738a43288840ed8@sha256:6a9bcd8337e513ee6738e145bca0a11284003ade110e5a3dfe9fa339aac4adb9",“appId”:1778390,“serviceId”:942848,“serviceName”:“pocfurukawa”,“imageId”:3353061,“releaseId”:1718715,“dependent”:0,“dockerImageId”:"sha256:1a4449ab18120e87e42f744c94f32b698c198840c0e7b2f19f61bcf588828282”}}
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName, stdinOpen
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:27 9d819d2 resin-supervisor[3717]: [warn] Ignoring unsupported or unknown compose fields: containerName
Mar 05 16:50:28 9d819d2 resin-supervisor[3717]: [event] Event: Service install {“service”:{“appId”:1778390,“serviceId”:942848,“serviceName”:“pocfurukawa”,“releaseId”:1720031}}
Mar 05 16:50:29 9d819d2 resin-supervisor[3717]: [error] Scheduling another update attempt in 900000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 404) no such container - sandbox 17287f6729d58f353bca622a7f3ddabe6cbdd4c6dc2667f58351a0168630d38f not found Steps:[“updateMetadata”,“updateMetadata”,“start”]
Mar 05 16:50:29 9d819d2 resin-supervisor[3717]: [error] at fn (/usr/src/app/dist/app.js:6:8488)
Mar 05 16:50:29 9d819d2 resin-supervisor[3717]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 404) no such container - sandbox 17287f6729d58f353bca622a7f3ddabe6cbdd4c6dc2667f58351a0168630d38f not found Steps:[“updateMetadata”,“updateMetadata”,“start”]
Mar 05 16:50:29 9d819d2 resin-supervisor[3717]: [error] at fn (/usr/src/app/dist/app.js:6:8488)
Mar 05 16:50:29 9d819d2 resin-supervisor[3717]: [event] Event: Service installed {“service”:{“appId”:1778390,“serviceId”:942848,“serviceName”:“pocfurukawa”,“releaseId”:1720031}}
Mar 05 16:50:29 9d819d2 resin-supervisor[3717]: [event] Event: Service start {“service”:{“appId”:1778390,“serviceId”:942848,“serviceName”:“pocfurukawa”,“releaseId”:1720031}}
Mar 05 16:50:31 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:50:31 9d819d2 resin-supervisor[3717]: [event] Event: Service started {“service”:{“appId”:1778390,“serviceId”:942848,“serviceName”:“pocfurukawa”,“releaseId”:1720031}}
Mar 05 16:50:31 9d819d2 resin-supervisor[3717]: [debug] Spawning journald with: chroot /mnt/root journalctl -a -S 2021-03-05 16:50:31 -o json CONTAINER_ID_FULL=92b361a4df5fcce624769b2c068a70708985b17e70fed97e11cacb1e4fc8a2dc
Mar 05 16:50:33 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:50:41 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:50:43 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:50:58 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:51:00 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:51:34 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:51:35 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:51:44 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:51:46 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:52:01 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:52:03 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:52:36 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:52:38 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:52:47 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:52:49 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:53:04 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:53:05 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:53:39 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:53:41 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:53:50 9d819d2 resin-supervisor[3717]: [api] GET /v1/healthy 200 - 23.659 ms
Mar 05 16:53:50 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:53:52 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:54:07 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:54:08 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:54:43 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:54:45 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:54:53 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:54:55 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:55:09 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:55:11 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:55:46 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:55:48 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:55:56 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:55:58 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:56:04 9d819d2 resin-supervisor[3717]: [debug] Attempting container log timestamp flush…
Mar 05 16:56:04 9d819d2 resin-supervisor[3717]: [debug] Container log timestamp flush complete
Mar 05 16:56:12 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:56:14 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:56:49 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:56:51 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945779,“serviceName”:“mqttsendbiturno”,“releaseId”:1720031}}
Mar 05 16:56:59 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:57:00 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:942852,“serviceName”:“mqttsendbihora”,“releaseId”:1718715}}
Mar 05 16:57:15 9d819d2 resin-supervisor[3717]: [event] Event: Service restart {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Mar 05 16:57:17 9d819d2 resin-supervisor[3717]: [event] Event: Service exit {“service”:{“appId”:1778390,“serviceId”:945782,“serviceName”:“mqttsendmanutencao”,“releaseId”:1718715}}
Hi, we have an open issue for a similar problem (Multiple services stuck on status Downloaded & not running · Issue #1612 · balena-io/balena-supervisor · GitHub) that is being investigated. I have linked that issue to this thread so we will update you once it is resolved. In that case, the workaround of restarting the supervisor will start all containers. Could yo u please check whether restarting the supervisor also works here so we can disambiguate whether they are the same or different problems?