Standard flow for testing a multi-container release on a single local BalenaOS device

Product support balenaOS
1

I recently got a working custom-domain openBalena system with several devices, and it is a huge improvement on the previous backup and remote update solution. One difficulty - how can I test that a new version works on a local test system, short of pushing to all of them with balena deploy?
Methods I looked at:

  • balena push to local device (I understand that this will both build and run on the local device): successfully builds and tags, but consistenctly hangs at “installing services” usign both local address and .local. A thread said that push only works with balenaCloud, though there might be some nuance that I am missing here.
  • pinning all production devices and letting the test device run the latest: pinning not available via balena-cli.
  • switching the dev device between a production and test application: could be messy. Would the layer cache survive the switch?
  • balena build: builds. Images are on the dev unit but not running and I have found no sign of a local docker-compose-like system to start all of them with the correct configuration.

So, starting to wander and would appreciate your insights. How do you approach deploying multi-container to a single device?

9

Hi there, happy to hear that openBalena is an improvement over what you were previously using. Just so I understand correctly, you are asking what the best way to test a new version of an application, without having to push it to all devices in the fleet?

If that’s the case, you could try using balena push <device IP address> to push to just that device. When pushing to just a single device ```balena push should be working. If this doesn’t work and gets stuck at installing services, could you please post the logs? Thanks.

18

Hello again - sorry just to clarify - in the case of failure could you please post the supervisor logs - these can be obtained through ssh’ing into the device host OS, and running journalctl -f -u resin-supervisor -n 1000.

Another thing - to use balena push <address>, the target device must be put into local mode. I would double check that this is the case - to set a device to local mode you can make an API call to set BALENA_SUPERVISOR_LOCAL_MODE variable for the device to 1.

1 Like
20

Thanks rcooke-warwick, it’s good to know where to focus. Getting the right log led to the answer while writing this response; still posting in case anyone needs a reminder to avoid unneeded docker-compose properties.

External build log was:

balena env add BALENA_SUPERVISOR_LOCAL_MODE 1 --device 8a30baa
balena push 192.168.0.27 --logs

[Info]    Starting build on device 192.168.0.27
[...]
[Build]   [tunnel] Successfully tagged u_tunnel:latest
[...]
[Build]   [mosquitto] Successfully tagged u_mosquitto:latest
[...]
[Build]   [maestro] Successfully tagged u_maestro:latest
[...]
[Build]   [wifi-connect] Successfully tagged local_image_wifi-connect:latest

[Live]    Waiting for device state to settle...
[Info]    Streaming device logs...
[Live]    Watching for file changes...
[Logs]    [5/28/2020, 8:35:37 AM] Creating network 'default'
[Logs]    [5/28/2020, 8:35:37 AM] Creating network 'backnet'
[Logs]    [5/28/2020, 8:35:38 AM] Installing service 'postgres sha256:6271ad73e6d1018b0be0e2c1b39feae69c9d9ee844d20a9add7d5ccb42fab558'
[Logs]    [5/28/2020, 8:35:38 AM] Installing service 'tunnel local_image_tunnel:latest'
[Logs]    [5/28/2020, 8:35:38 AM] Installing service 'wifi-connect sha256:0da722c59ff799ec639f3412fcf494d624b452a50dbe46f00d48d2985dfe6de2'
[Logs]    [5/28/2020, 8:35:38 AM] Installing service 'maestro local_image_maestro:latest'
[Logs]    [5/28/2020, 8:35:38 AM] Installed service 'postgres sha256:6271ad73e6d1018b0be0e2c1b39feae69c9d9ee844d20a9add7d5ccb42fab558'
[Logs]    [5/28/2020, 8:35:38 AM] Starting service 'postgres sha256:6271ad73e6d1018b0be0e2c1b39feae69c9d9ee844d20a9add7d5ccb42fab558'
[Logs]    [5/28/2020, 8:35:38 AM] Installed service 'wifi-connect sha256:0da722c59ff799ec639f3412fcf494d624b452a50dbe46f00d48d2985dfe6de2'
[Logs]    [5/28/2020, 8:35:38 AM] Starting service 'wifi-connect sha256:0da722c59ff799ec639f3412fcf494d624b452a50dbe46f00d48d2985dfe6de2'
[Logs]    [5/28/2020, 8:35:40 AM] Started service 'wifi-connect sha256:0da722c59ff799ec639f3412fcf494d624b452a50dbe46f00d48d2985dfe6de2'
[Logs]    [5/28/2020, 8:35:41 AM] Starting service 'postgres sha256:6271ad73e6d1018b0be0e2c1b39feae69c9d9ee844d20a9add7d5ccb42fab558'
[Logs]    [5/28/2020, 8:35:45 AM] Started service 'postgres sha256:6271ad73e6d1018b0be0e2c1b39feae69c9d9ee844d20a9add7d5ccb42fab558'
[Logs]    [5/28/2020, 8:35:45 AM] Installing service 'maestro local_image_maestro:latest'
[Logs]    [5/28/2020, 8:35:45 AM] Installing service 'tunnel local_image_tunnel:latest'
[Logs]    [5/28/2020, 8:35:45 AM] [postgres] PostgreSQL Database directory appears to contain a database; Skipping initialization
*[ ... other logs from now-running postgres container]*
[Logs]    [5/28/2020, 8:35:48 AM] Installing service 'mosquitto local_image_mosquitto:latest'
[Logs]    [5/28/2020, 8:35:48 AM] Installing service 'tunnel local_image_tunnel:latest'
[Logs]    [5/28/2020, 8:35:48 AM] Installing service 'maestro local_image_maestro:latest'
[Logs]    [5/28/2020, 8:35:52 AM] Installing service 'maestro local_image_maestro:latest'
[Logs]    [5/28/2020, 8:35:52 AM] Installing service 'mosquitto local_image_mosquitto:latest'
[Logs]    [5/28/2020, 8:35:52 AM] Installing service 'tunnel local_image_tunnel:latest'
[Logs]    [5/28/2020, 8:35:55 AM] [wifi-connect] *[Network name - wifi-connect is running too]*
[Logs]    [5/28/2020, 8:35:55 AM] [wifi-connect] Skipping WiFi Connect
[Logs]    [5/28/2020, 8:36:01 AM] Installing service 'tunnel local_image_tunnel:latest'
[Logs]    [5/28/2020, 8:36:01 AM] Installing service 'mosquitto local_image_mosquitto:latest'
[Logs]    [5/28/2020, 8:36:01 AM] Installing service 'maestro local_image_maestro:latest'
[Logs]    [5/28/2020, 8:36:17 AM] Installing service 'mosquitto local_image_mosquitto:latest'
[Logs]    [5/28/2020, 8:36:18 AM] Installing service 'maestro local_image_maestro:latest'
[Logs]    [5/28/2020, 8:36:18 AM] Installing service 'tunnel local_image_tunnel:latest'
[Logs]    [5/28/2020, 8:36:50 AM] Installing service 'mosquitto local_image_mosquitto:latest'
[Logs]    [5/28/2020, 8:36:50 AM] Installing service 'maestro local_image_maestro:latest'
[Logs]    [5/28/2020, 8:36:50 AM] Installing service 'tunnel local_image_tunnel:latest'
[Logs]    [5/28/2020, 8:37:56 AM] Installing service 'tunnel local_image_tunnel:latest'
[Logs]    [5/28/2020, 8:37:56 AM] Installing service 'maestro local_image_maestro:latest'
[Logs]    [5/28/2020, 8:37:56 AM] Installing service 'mosquitto local_image_mosquitto:latest'
*[... repeats every few minutes]*

Corresponding

with redundant uuid/proc name cleared:

May 28 15:36:17 8a30baa resin-supervisor[1405]: [info]    Applying target state
May 28 15:36:17: [warn]    Ignoring unsupported or unknown compose fields: containerName
May 28 15:36:17: [warn]    Ignoring unsupported or unknown compose fields: containerName
May 28 15:36:17: [warn]    Ignoring unsupported or unknown compose fields: containerName
May 28 15:36:17: [warn]    Ignoring unsupported or unknown compose fields: containerName
May 28 15:36:17: [event]   Event: Service install {"service":{"appId":1,"serviceId":2,"serviceName":"mosquitto","releaseId":1}}
May 28 15:36:17: [error]   Scheduling another update attempt in 32000ms due to failure:  Error: Failed to apply state transition steps. (HTTP code 404) no such container - No such image: local_image_mosquitto:latest  Steps:["start","start","start"]
May 28 15:36:17: [error]         at /usr/src/app/dist/app.js:614:16375
May 28 15:36:17: [error]       at c (/usr/src/app/dist/app.js:9:77523)
May 28 15:36:17: [error]       at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
May 28 15:36:17: [error]       at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
May 28 15:36:17: [error]       at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
May 28 15:36:17: [error]       at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
May 28 15:36:17: [error]       at d (/usr/src/app/dist/app.js:312:329886)
May 28 15:36:17: [error]       at p (/usr/src/app/dist/app.js:312:329825)
May 28 15:36:17: [error]       at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
May 28 15:36:17: [error]       at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
May 28 15:36:17: [error]       at runCallback (timers.js:705:18)
May 28 15:36:17: [error]       at tryOnImmediate (timers.js:676:5)
May 28 15:36:17: [error]       at processImmediate (timers.js:658:5)
May 28 15:36:17: [error]   Device state apply error Error: Failed to apply state transition steps. (HTTP code 404) no such container - No such image: local_image_mosquitto:latest  Steps:["start","start","start"]
May 28 15:36:17: [error]         at /usr/src/app/dist/app.js:614:16375
May 28 15:36:17: [error]       at c (/usr/src/app/dist/app.js:9:77523)
May 28 15:36:17: [error]       at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
May 28 15:36:17: [error]       at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
May 28 15:36:17: [error]       at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
May 28 15:36:17: [error]       at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
May 28 15:36:17: [error]       at d (/usr/src/app/dist/app.js:312:329886)
May 28 15:36:17: [error]       at p (/usr/src/app/dist/app.js:312:329825)
May 28 15:36:17: [error]       at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
May 28 15:36:17: [error]       at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
May 28 15:36:17: [error]       at runCallback (timers.js:705:18)
May 28 15:36:17: [error]       at tryOnImmediate (timers.js:676:5)
May 28 15:36:17: [error]       at processImmediate (timers.js:658:5)
May 28 15:36:18: [event]   Event: Service install {"service":{"appId":1,"serviceId":3,"serviceName":"maestro","releaseId":1}}
May 28 15:36:18: [event]   Event: Service install {"service":{"appId":1,"serviceId":4,"serviceName":"tunnel","releaseId":1}}
May 28 15:36:18: [api]     GET /v2/state/status 200 - 204.955 ms
*[more v2 200s]*
May 28 15:36:49: [info]    Applying target state
May 28 15:36:50: [warn]    Ignoring unsupported or unknown compose fields: containerName
May 28 15:36:50: [warn]    Ignoring unsupported or unknown compose fields: containerName
May 28 15:36:50: [warn]    Ignoring unsupported or unknown compose fields: containerName
May 28 15:36:50: [warn]    Ignoring unsupported or unknown compose fields: containerName
May 28 15:36:50: [event]   Event: Service install {"service":{"appId":1,"serviceId":2,"serviceName":"mosquitto","releaseId":1}}
May 28 15:36:50: [event]   Event: Service install {"service":{"appId":1,"serviceId":3,"serviceName":"maestro","releaseId":1}}
May 28 15:36:50: [error]   Scheduling another update attempt in 64000ms due to failure:  Error: Failed to apply state transition steps. (HTTP code 404) no such container - No such image: local_image_mosquitto:latest  Steps:["start","start","start"]
May 28 15:36:50: [error]         at /usr/src/app/dist/app.js:614:16375
May 28 15:36:50: [error]       at c (/usr/src/app/dist/app.js:9:77523)
May 28 15:36:50: [error]       at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
May 28 15:36:50: [error]       at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
May 28 15:36:50: [error]       at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
May 28 15:36:50: [error]       at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
May 28 15:36:50: [error]       at d (/usr/src/app/dist/app.js:312:329886)
May 28 15:36:50: [error]       at p (/usr/src/app/dist/app.js:312:329825)
May 28 15:36:50: [error]       at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
May 28 15:36:50: [error]       at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
May 28 15:36:50: [error]       at runCallback (timers.js:705:18)
May 28 15:36:50: [error]       at tryOnImmediate (timers.js:676:5)
May 28 15:36:50: [error]       at processImmediate (timers.js:658:5)
May 28 15:36:50: [event]   Event: Service install {"service":{"appId":1,"serviceId":4,"serviceName":"tunnel","releaseId":1}}
May 28 15:36:50: [error]   Device state apply error Error: Failed to apply state transition steps. (HTTP code 404) no such container - No such image: local_image_mosquitto:latest  Steps:["start","start","start"]
May 28 15:36:50: [error]         at /usr/src/app/dist/app.js:614:16375
May 28 15:36:50: [error]       at c (/usr/src/app/dist/app.js:9:77523)
May 28 15:36:50: [error]       at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
May 28 15:36:50: [error]       at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
May 28 15:36:50: [error]       at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
May 28 15:36:50: [error]       at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
May 28 15:36:50: [error]       at d (/usr/src/app/dist/app.js:312:329886)
May 28 15:36:50: [error]       at p (/usr/src/app/dist/app.js:312:329825)
May 28 15:36:50: [error]       at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
May 28 15:36:50: [error]       at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
May 28 15:36:50: [error]       at runCallback (timers.js:705:18)
May 28 15:36:50: [error]       at tryOnImmediate (timers.js:676:5)
May 28 15:36:50: [error]       at processImmediate (timers.js:658:5)
May 28 15:36:50: [api]     GET /v2/state/status 200 - 337.490 ms      
*[more v2 200s]*
May 28 15:37:53: [api]     GET /v2/state/status 200 - 231.489 ms
May 28 15:37:54: [info]    Applying target state
May 28 15:37:54: [warn]    Ignoring unsupported or unknown compose fields: containerName
May 28 15:37:54: [warn]    Ignoring unsupported or unknown compose fields: containerName
May 28 15:37:54: [warn]    Ignoring unsupported or unknown compose fields: containerName
May 28 15:37:54: [warn]    Ignoring unsupported or unknown compose fields: containerName
May 28 15:37:54: [event]   Event: Service install {"service":{"appId":1,"serviceId":4,"serviceName":"tunnel","releaseId":1}}
May 28 15:37:54: [event]   Event: Service install {"service":{"appId":1,"serviceId":3,"serviceName":"maestro","releaseId":1}}
May 28 15:37:55: [error]   Scheduling another update attempt in 128000ms due to failure:  Error: Failed to apply state transition steps. (HTTP code 404) no such container - No such image: local_image_tunnel:latest  Steps:["start","start","start"]
May 28 15:37:55: [error]         at /usr/src/app/dist/app.js:614:16375
May 28 15:37:55: [error]       at c (/usr/src/app/dist/app.js:9:77523)
May 28 15:37:55: [error]       at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
May 28 15:37:55: [error]       at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
May 28 15:37:55: [error]       at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
May 28 15:37:55: [error]       at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
May 28 15:37:55: [error]       at d (/usr/src/app/dist/app.js:312:329886)
May 28 15:37:55: [error]       at p (/usr/src/app/dist/app.js:312:329825)
May 28 15:37:55: [error]       at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
May 28 15:37:55: [error]       at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
May 28 15:37:55: [error]       at runCallback (timers.js:705:18)
May 28 15:37:55: [error]       at tryOnImmediate (timers.js:676:5)
May 28 15:37:55: [error]       at processImmediate (timers.js:658:5)
May 28 15:37:55: [event]   Event: Service install {"service":{"appId":1,"serviceId":2,"serviceName":"mosquitto","releaseId":1}}
May 28 15:37:55: [error]   Device state apply error Error: Failed to apply state transition steps. (HTTP code 404) no such container - No such image: local_image_tunnel:latest  Steps:["start","start","start"]
May 28 15:37:55: [error]         at /usr/src/app/dist/app.js:614:16375
May 28 15:37:55: [error]       at c (/usr/src/app/dist/app.js:9:77523)
May 28 15:37:55: [error]       at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
May 28 15:37:55: [error]       at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
May 28 15:37:55: [error]       at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
May 28 15:37:55: [error]       at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
May 28 15:37:55: [error]       at d (/usr/src/app/dist/app.js:312:329886)
May 28 15:37:55: [error]       at p (/usr/src/app/dist/app.js:312:329825)
May 28 15:37:55: [error]       at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
May 28 15:37:55: [error]       at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
May 28 15:37:55: [error]       at runCallback (timers.js:705:18)
May 28 15:37:55: [error]       at tryOnImmediate (timers.js:676:5)
May 28 15:37:55: [error]       at processImmediate (timers.js:658:5)
May 28 15:37:56: [api]     GET /v2/state/status 200 - 205.830 ms
*[more v2 200s]*
May 28 15:38:26: [api]     GET /v2/state/status 200 - 197.453 ms
May 28 15:38:26: [api]     GET /v1/healthy 200 - 17.344 ms
*[more v2 200s]*

So, containers based on a template and an outside image both ran fine, but the images for the ones built with Dockerfile were not found, because it expects an image with a different prefix. I deleted the offending docker-compose.yml container_name entries (they were like u_mosquitto, now the u_ is only in the service names), but the pattern was still:

Successfully tagged u_mosquitto:latest
Installing service 'mosquitto local_image_mosquitto:latest'
No such image: local_image_mosquitto:latest  Steps:["start","start","start"]

Turns out that it was another remnant of the old method: docker-compose image properties on the custom images, used before openBalena automated the image download process. I removed that and all pushed containers are running now. Thank you!

21

Hey, glad to see that you figured it out. Let us know if you get stuck or you need further assistance. Have a nice day!

1 Like