Hi Balena team,
i am currently experiencing a strange problem with all my devices(rpi3 and cm4). It seems like they are only able to download and run the containers the first time.
If there is any other release uploaded to the server or i manually change the release of the device the supervisor gets stuck.
Sometimes while trying to delete the containers, sometimes after deleting them, or after downloading the new ones.
What i have mainly observed is that the supervisor starts complained about unmanaged or anonymous Volumes it finds
Here are some example logs from one of the devices
[event] Event: Image removed {"image":{"name":"registry2.**.**v2/d8a964a23309c2afe1806785864c7acb@sha256:71b1a7be84ee0687aa80f5c56e5c82c55480a513149c8e420c5ca584109bb908","appId":6,"serviceId":47,"serviceName":"InfluxDB","imageId":298,"releaseId":48,"appUuid":"fa9e370bcdb642fdba977281356b71c9","commit":"193c63301539c172f89c5a71ad2722a8"}}
[event] Event: Image removed {"image":{"name":"registry2.**.**/v2/5c48e1fc0cb04d1479842ac753d9b7c7@sha256:210aa54450ec6636993ca29f1f29116047d4134d671667f05edcefd216b4ab36","appId":6,"serviceId":46,"serviceName":"Grafana","imageId":300,"releaseId":48,"appUuid":"fa9e370bcdb642fdba977281356b71c9","commit":"193c63301539c172f89c5a71ad2722a8"}}
[info] Reported current state to the cloud
[event] Event: Image removed {"image":{"name":"registry2.**.**/v2/5e98d40b085fa32cba4cb90684a99732@sha256:180ce9bce2709346d2fc4fc1b269ecc050375f55b968c190d004a4d3d5fbf028","appId":6,"serviceId":48,"serviceName":"Node-Red","imageId":301,"releaseId":48,"appUuid":"fa9e370bcdb642fdba977281356b71c9","commit":"193c63301539c172f89c5a71ad2722a8"}}
[event] Event: Image removed {"image":{"name":"registry2.**.**/v2/ef804a2cd4e9732b33bdce65d51747bb@sha256:80dc9ed286666bb34cce38a5570126d4ea7fe62ff4f37fa94132281d35ca4acf","appId":6,"serviceId":49,"serviceName":"****","imageId":302,"releaseId":48,"appUuid":"fa9e370bcdb642fdba977281356b71c9","commit":"193c63301539c172f89c5a71ad2722a8"}}
[event] Event: Image removed {"image":{"name":"registry2.**.**/v2/d1a9b83b2dbb630651d1f5f0a740dc8c@sha256:082ed330a51354597da9de3731791f55152fbf5b4f6c29c1196d5a19c8a7f717","appId":6,"serviceId":44,"serviceName":"Valkey","imageId":297,"releaseId":48,"appUuid":"fa9e370bcdb642fdba977281356b71c9","commit":"193c63301539c172f89c5a71ad2722a8"}}
[event] Event: Image removal {"image":{"name":"registry2.**.**/v2/e6938cb89a471030ead4b4f3ca117ba5@sha256:0beed0e222e19177084a456f6b2e7c5e00965f741c3fd49c601c976068d1897e","appId":6,"serviceId":45,"serviceName":"Mosquitto","imageId":299,"releaseId":48,"appUuid":"fa9e370bcdb642fdba977281356b71c9","commit":"193c63301539c172f89c5a71ad2722a8"}}
[event] Event: Image removed {"image":{"name":"registry2.**.**/v2/e6938cb89a471030ead4b4f3ca117ba5@sha256:0beed0e222e19177084a456f6b2e7c5e00965f741c3fd49c601c976068d1897e","appId":6,"serviceId":45,"serviceName":"Mosquitto","imageId":299,"releaseId":48,"appUuid":"fa9e370bcdb642fdba977281356b71c9","commit":"193c63301539c172f89c5a71ad2722a8"}}
[event] Event: Network removal {}
[event] Event: Network creation {}
[event] Event: Release update locks {"appId":"6"}
[debug] Finished applying target state
[success] Device state apply success
[api] GET /v1/healthy 200 - 7.266 ms
[info] Reported current state to the cloud
[api] GET /v1/healthy 200 - 5.315 ms
[info] Reported current state to the cloud
[info] Applying target state
[event] Event: Network removal {}
[event] Event: Docker image download {"image":{"name":"registry2.**.**/v2/e6938cb89a471030ead4b4f3ca117ba5@sha256:0beed0e222e19177084a456f6b2e7c5e00965f741c3fd49c601c976068d1897e","appId":6,"appUuid":"fa9e370bcdb642fdba977281356b71c9","serviceId":45,"serviceName":"Mosquitto","imageId":299,"releaseId":48,"commit":"193c63301539c172f89c5a71ad2722a8"}}
[event] Event: Docker image download {"image":{"name":"registry2.**.**/v2/5e98d40b085fa32cba4cb90684a99732@sha256:180ce9bce2709346d2fc4fc1b269ecc050375f55b968c190d004a4d3d5fbf028","appId":6,"appUuid":"fa9e370bcdb642fdba977281356b71c9","serviceId":48,"serviceName":"Node-Red","imageId":301,"releaseId":48,"commit":"193c63301539c172f89c5a71ad2722a8"}}
[event] Event: Docker image download {"image":{"name":"registry2.**.**/v2/d1a9b83b2dbb630651d1f5f0a740dc8c@sha256:082ed330a51354597da9de3731791f55152fbf5b4f6c29c1196d5a19c8a7f717","appId":6,"appUuid":"fa9e370bcdb642fdba977281356b71c9","serviceId":44,"serviceName":"Valkey","imageId":297,"releaseId":48,"commit":"193c63301539c172f89c5a71ad2722a8"}}
[event] Event: Docker image download {"image":{"name":"registry2.**.**/v2/5cb9a94b512bd95b79888b4e69c413b0@sha256:6d802f90258d3230279924609f43226d927a57770309ad915ca20d9bc2a7c735","appId":6,"appUuid":"fa9e370bcdb642fdba977281356b71c9","serviceId":43,"serviceName":"java-jre-alpine","imageId":296,"releaseId":48,"commit":"193c63301539c172f89c5a71ad2722a8"}}
[event] Event: Docker image download {"image":{"name":"registry2.**.**/v2/d8a964a23309c2afe1806785864c7acb@sha256:71b1a7be84ee0687aa80f5c56e5c82c55480a513149c8e420c5ca584109bb908","appId":6,"appUuid":"fa9e370bcdb642fdba977281356b71c9","serviceId":47,"serviceName":"InfluxDB","imageId":298,"releaseId":48,"commit":"193c63301539c172f89c5a71ad2722a8"}}
[event] Event: Docker image download {"image":{"name":"registry2.**.**/v2/5c48e1fc0cb04d1479842ac753d9b7c7@sha256:210aa54450ec6636993ca29f1f29116047d4134d671667f05edcefd216b4ab36","appId":6,"appUuid":"fa9e370bcdb642fdba977281356b71c9","serviceId":46,"serviceName":"Grafana","imageId":300,"releaseId":48,"commit":"193c63301539c172f89c5a71ad2722a8"}}
[event] Event: Docker image download {"image":{"name":"registry2.**.**/v2/ef804a2cd4e9732b33bdce65d51747bb@sha256:80dc9ed286666bb34cce38a5570126d4ea7fe62ff4f37fa94132281d35ca4acf","appId":6,"appUuid":"fa9e370bcdb642fdba977281356b71c9","serviceId":49,"serviceName":"DC-Core-213","imageId":302,"releaseId":48,"commit":"193c63301539c172f89c5a71ad2722a8"}}
[info] Reported current state to the cloud
[info] Reported current state to the cloud
[info] Reported current state to the cloud
...
[event] Event: Take update locks {"appId":"6","force":false,"services":["Mosquitto","Node-Red","Valkey","java-jre-alpine","InfluxDB","Grafana","DC-Core-213"]}
[event] Event: Network creation {}
[event] Event: Service install {"service":{"appId":6,"serviceId":44,"serviceName":"Valkey","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[event] Event: Service install {"service":{"appId":6,"serviceId":43,"serviceName":"java-jre-alpine","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[event] Event: Service install {"service":{"appId":6,"serviceId":47,"serviceName":"InfluxDB","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[event] Event: Service installed {"service":{"appId":6,"serviceId":43,"serviceName":"java-jre-alpine","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[event] Event: Service start {"service":{"appId":6,"serviceId":43,"serviceName":"java-jre-alpine","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[event] Event: Service installed {"service":{"appId":6,"serviceId":44,"serviceName":"Valkey","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[event] Event: Service start {"service":{"appId":6,"serviceId":44,"serviceName":"Valkey","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[event] Event: Service installed {"service":{"appId":6,"serviceId":47,"serviceName":"InfluxDB","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[event] Event: Service start {"service":{"appId":6,"serviceId":47,"serviceName":"InfluxDB","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[event] Event: Service started {"service":{"appId":6,"serviceId":43,"serviceName":"java-jre-alpine","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[event] Event: Service started {"service":{"appId":6,"serviceId":44,"serviceName":"Valkey","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[event] Event: Service started {"service":{"appId":6,"serviceId":47,"serviceName":"InfluxDB","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[event] Event: Service exit {"service":{"appId":6,"serviceId":43,"serviceName":"java-jre-alpine","commit":"193c63301539c172f89c5a71ad2722a8","releaseId":48}}
[debug] Found unmanaged or anonymous Volume: 6bea8ea94ffd736d2e656d696c958aef7aa93f6167b4bd455a67529b1b0d96ca
[debug] Found unmanaged or anonymous Volume: 2fe124f7602c141849508c3bbec814e5004405119420820db248b400311b28ac
[debug] Found unmanaged or anonymous Volume: 6bea8ea94ffd736d2e656d696c958aef7aa93f6167b4bd455a67529b1b0d96ca
[debug] Found unmanaged or anonymous Volume: 2fe124f7602c141849508c3bbec814e5004405119420820db248b400311b28ac
[debug] Found unmanaged or anonymous Volume: 6bea8ea94ffd736d2e656d696c958aef7aa93f6167b4bd455a67529b1b0d96ca
[debug] Found unmanaged or anonymous Volume: 2fe124f7602c141849508c3bbec814e5004405119420820db248b400311b28ac
[debug] Found unmanaged or anonymous Volume: 2fe124f7602c141849508c3bbec814e5004405119420820db248b400311b28ac
[debug] Found unmanaged or anonymous Volume: 6bea8ea94ffd736d2e656d696c958aef7aa93f6167b4bd455a67529b1b0d96ca
[info] Reported current state to the cloud
[debug] Found unmanaged or anonymous Volume: 6bea8ea94ffd736d2e656d696c958aef7aa93f6167b4bd455a67529b1b0d96ca
[debug] Found unmanaged or anonymous Volume: 2fe124f7602c141849508c3bbec814e5004405119420820db248b400311b28ac
[debug] Found unmanaged or anonymous Volume: 2fe124f7602c141849508c3bbec814e5004405119420820db248b400311b28ac
[debug] Found unmanaged or anonymous Volume: 6bea8ea94ffd736d2e656d696c958aef7aa93f6167b4bd455a67529b1b0d96ca
[debug] Found unmanaged or anonymous Volume: 2fe124f7602c141849508c3bbec814e5004405119420820db248b400311b28ac
[debug] Found unmanaged or anonymous Volume: 6bea8ea94ffd736d2e656d696c958aef7aa93f6167b4bd455a67529b1b0d96
.... this goes on and on
The device will then just end up with all its containers deleted (as shown by running balena ps, inside the device) but never starting setting up the new containers.
I am currently using
balenaOS: balenaOS 6.1.24+rev4
supervisor:16.10.3
openbalena:4.1.* (the latests)