Raspberry Pi v1 app was running well until I updated the balena os and supervisor to the latest versions. Now the services all start up, then exit without logging anything. Is this a known issue? Is there a way to downgrade the OS/supervisor?
Hi there, thanks for reaching out. What versions of the OS + supervisor did you update from-> to? We don’t support OS downgrades unfortunately - but we can try to get more information as to why this might be happening for you.
steps you could try to get more information:
- in the dashboard, you can go to the diagnostics tab, and then run healthchecks - see if any fail, and let us know
- you could ssh into the host OS of the device, either through the dashboard terminal or via the CLI, and run
journalctl --no-pager -a -n 1000 -u balena.service- and see if there is any additional logging before the services exit that looks suspicious
In the diagnostics the check_container_engine check failed (3 restarts, maybe crashlooping). check_networking also failed (could not communicate with registry2.balena-cloud.com for auth) and check_service_restarts (some services are restarting unexpectedly - restart count 1)
In the system log I’m seeing a periodic error about “Scheduling another update attempt due to failure: Error: Failed to apply state transition steps: Bind for 0.0.0.0:80 port is already allocated”.
Hi @disparlure, this sounds like an issue we’ve encountered before. See this GitHub issue: Port already in use, because proxy keeps binding to the wrong container IP · Issue #272 · balena-os/balena-engine · GitHub. Feel free to follow it or ping in it to move it up the queue.
Is your device still erroring? Could you provide the UUID and support access for up to 1 week please, so that support agents can take a look?
Thanks,
Christina
The fact that there are networking issues unrelated to this linked Engine issue though (i.e. couldn’t communicate with registry), indicate that there may be something else going on. Please do paste logs as mentioned by my colleague above when you get a chance, to aid support agents in faster investigation.
These logs were captured while trying the fix described in issue 272 above.
Jan 06 15:42:24 9db5f18 5702a46ac234[3088]: [info] Supervisor v12.11.22 starting up…
Jan 06 15:42:28 9db5f18 5702a46ac234[3088]: [info] Setting host to discoverable
Jan 06 15:42:28 9db5f18 5702a46ac234[3088]: [warn] Invalid firewall mode: . Reverting to state: off
Jan 06 15:42:28 9db5f18 5702a46ac234[3088]: [info] Applying firewall mode: off
Jan 06 15:42:29 9db5f18 5702a46ac234[3088]: [debug] Starting systemd unit: avahi-daemon.service
Jan 06 15:42:29 9db5f18 5702a46ac234[3088]: [debug] Starting systemd unit: avahi-daemon.socket
Jan 06 15:42:29 9db5f18 5702a46ac234[3088]: [debug] Starting logging infrastructure
Jan 06 15:42:30 9db5f18 5702a46ac234[3088]: [info] Starting firewall
Jan 06 15:42:30 9db5f18 5702a46ac234[3088]: [debug] Performing database cleanup for container log timestamps
Jan 06 15:42:31 9db5f18 balenad[2740]: Parent process exited
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [info] Previous engine snapshot was not stored. Skipping cleanup.
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [debug] Handling of local mode switch is completed
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [success] Firewall mode applied
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [debug] Starting api binder
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [info] API Binder bound to: https://api.balena-cloud.com/v6/
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [event] Event: Supervisor start {}
Jan 06 15:42:34 9db5f18 5702a46ac234[3088]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [debug] Connectivity check enabled: true
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [debug] Starting periodic check for IP addresses
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [info] Reporting initial state, supervisor version and API info
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [debug] Skipping preloading
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [info] Starting API server
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] Supervisor API successfully started on port 48484
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [debug] VPN status path exists.
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [debug] Ensuring device is provisioned
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] VPN connection is active.
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] Waiting for connectivity…
Jan 06 15:42:37 9db5f18 5702a46ac234[3088]: [debug] Starting current state report
Jan 06 15:42:37 9db5f18 5702a46ac234[3088]: [debug] Starting target state poll
Jan 06 15:42:37 9db5f18 5702a46ac234[3088]: [debug] Spawning journald with: chroot /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
Jan 06 15:42:45 9db5f18 5702a46ac234[3088]: [event] Event: Service start {“service”:{“appId”:1775425,“serviceId”:841874,“serviceName”:“influxdb”,“commit”:“024a5f2e64c0c4fdfa4a712585ae635b”,“releaseId”:1994829}}
Jan 06 15:42:46 9db5f18 5702a46ac234[3088]: [info] Internet Connectivity: OK
Jan 06 15:42:49 9db5f18 balenad[3088]: time=“2022-01-06T15:42:49.416886875Z” level=error msg=“stream copy error: reading from a closed fifo”
Jan 06 15:42:52 9db5f18 balenad[3088]: time=“2022-01-06T15:42:52.181595625Z” level=error msg=“00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container”
Jan 06 15:42:52 9db5f18 balenad[3088]: time=“2022-01-06T15:42:52.188806554Z” level=error msg=“Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown”
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] Updating failed, but there’s another update scheduled immediately: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:42:58 9db5f18 5702a46ac234[3088]: [event] Event: Service start {“service”:{“appId”:1775425,“serviceId”:841874,“serviceName”:“influxdb”,“commit”:“024a5f2e64c0c4fdfa4a712585ae635b”,“releaseId”:1994829}}
Jan 06 15:43:03 9db5f18 balenad[3088]: time=“2022-01-06T15:43:03.460527456Z” level=error msg=“stream copy error: reading from a closed fifo”
Jan 06 15:43:06 9db5f18 balenad[3088]: time=“2022-01-06T15:43:06.311130360Z” level=error msg=“00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container”
Jan 06 15:43:06 9db5f18 balenad[3088]: time=“2022-01-06T15:43:06.318270289Z” level=error msg=“Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown”
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 2000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:08 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:43:13 9db5f18 5702a46ac234[3088]: [event] Event: Service start {“service”:{“appId”:1775425,“serviceId”:841874,“serviceName”:“influxdb”,“commit”:“024a5f2e64c0c4fdfa4a712585ae635b”,“releaseId”:1994829}}
Jan 06 15:43:17 9db5f18 balenad[3088]: time=“2022-01-06T15:43:17.118199842Z” level=error msg=“stream copy error: reading from a closed fifo”
Jan 06 15:43:19 9db5f18 balenad[3088]: time=“2022-01-06T15:43:19.954881883Z” level=error msg=“00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container”
Jan 06 15:43:19 9db5f18 balenad[3088]: time=“2022-01-06T15:43:19.960671825Z” level=error msg=“Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown”
Jan 06 15:43:19 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 4000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks ()
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks ()
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:24 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:43:28 9db5f18 5702a46ac234[3088]: [event] Event: Service start {“service”:{“appId”:1775425,“serviceId”:841874,“serviceName”:“influxdb”,“commit”:“024a5f2e64c0c4fdfa4a712585ae635b”,“releaseId”:1994829}}
Jan 06 15:43:32 9db5f18 balenad[3088]: time=“2022-01-06T15:43:32.300883196Z” level=error msg=“stream copy error: reading from a closed fifo”
Jan 06 15:43:35 9db5f18 balenad[3088]: time=“2022-01-06T15:43:35.220046424Z” level=error msg=“00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container”
Jan 06 15:43:35 9db5f18 balenad[3088]: time=“2022-01-06T15:43:35.227147354Z” level=error msg=“Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown”
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 8000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks ()
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks ()
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:43 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:43:47 9db5f18 5702a46ac234[3088]: [event] Event: Service start {“service”:{“appId”:1775425,“serviceId”:841874,“serviceName”:“influxdb”,“commit”:“024a5f2e64c0c4fdfa4a712585ae635b”,“releaseId”:1994829}}
Jan 06 15:43:50 9db5f18 balenad[3088]: time=“2022-01-06T15:43:50.452489288Z” level=error msg=“stream copy error: reading from a closed fifo”
Jan 06 15:43:53 9db5f18 balenad[3088]: time=“2022-01-06T15:43:53.098094212Z” level=error msg=“00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container”
Jan 06 15:43:53 9db5f18 balenad[3088]: time=“2022-01-06T15:43:53.106892125Z” level=error msg=“Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown”
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 16000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks ()
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks ()
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:44:09 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:44:14 9db5f18 5702a46ac234[3088]: [event] Event: Service start {“service”:{“appId”:1775425,“serviceId”:841874,“serviceName”:“influxdb”,“commit”:“024a5f2e64c0c4fdfa4a712585ae635b”,“releaseId”:1994829}}
The following logs were captured after trying the fix linked above:
Jan 06 15:42:24 9db5f18 5702a46ac234[3088]: [info] Supervisor v12.11.22 starting up…
Jan 06 15:42:28 9db5f18 5702a46ac234[3088]: [info] Setting host to discoverable
Jan 06 15:42:28 9db5f18 5702a46ac234[3088]: [warn] Invalid firewall mode: . Reverting to state: off
Jan 06 15:42:28 9db5f18 5702a46ac234[3088]: [info] Applying firewall mode: off
Jan 06 15:42:29 9db5f18 5702a46ac234[3088]: [debug] Starting systemd unit: avahi-daemon.service
Jan 06 15:42:29 9db5f18 5702a46ac234[3088]: [debug] Starting systemd unit: avahi-daemon.socket
Jan 06 15:42:29 9db5f18 5702a46ac234[3088]: [debug] Starting logging infrastructure
Jan 06 15:42:30 9db5f18 5702a46ac234[3088]: [info] Starting firewall
Jan 06 15:42:30 9db5f18 5702a46ac234[3088]: [debug] Performing database cleanup for container log timestamps
Jan 06 15:42:31 9db5f18 balenad[2740]: Parent process exited
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [info] Previous engine snapshot was not stored. Skipping cleanup.
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [debug] Handling of local mode switch is completed
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [success] Firewall mode applied
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [debug] Starting api binder
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [info] API Binder bound to: https://api.balena-cloud.com/v6/
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [event] Event: Supervisor start {}
Jan 06 15:42:34 9db5f18 5702a46ac234[3088]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [debug] Connectivity check enabled: true
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [debug] Starting periodic check for IP addresses
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [info] Reporting initial state, supervisor version and API info
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [debug] Skipping preloading
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [info] Starting API server
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] Supervisor API successfully started on port 48484
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [debug] VPN status path exists.
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [debug] Ensuring device is provisioned
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] VPN connection is active.
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] Waiting for connectivity…
Jan 06 15:42:37 9db5f18 5702a46ac234[3088]: [debug] Starting current state report
Jan 06 15:42:37 9db5f18 5702a46ac234[3088]: [debug] Starting target state poll
Jan 06 15:42:37 9db5f18 5702a46ac234[3088]: [debug] Spawning journald with: chroot /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
Jan 06 15:42:45 9db5f18 5702a46ac234[3088]: [event] Event: Service start {“service”:{“appId”:1775425,“serviceId”:841874,“serviceName”:“influxdb”,“commit”:“024a5f2e64c0c4fdfa4a712585ae635b”,“releaseId”:1994829}}
Jan 06 15:42:46 9db5f18 5702a46ac234[3088]: [info] Internet Connectivity: OK
Jan 06 15:42:49 9db5f18 balenad[3088]: time=“2022-01-06T15:42:49.416886875Z” level=error msg=“stream copy error: reading from a closed fifo”
Jan 06 15:42:52 9db5f18 balenad[3088]: time=“2022-01-06T15:42:52.181595625Z” level=error msg=“00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container”
Jan 06 15:42:52 9db5f18 balenad[3088]: time=“2022-01-06T15:42:52.188806554Z” level=error msg=“Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown”
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] Updating failed, but there’s another update scheduled immediately: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:42:58 9db5f18 5702a46ac234[3088]: [event] Event: Service start {“service”:{“appId”:1775425,“serviceId”:841874,“serviceName”:“influxdb”,“commit”:“024a5f2e64c0c4fdfa4a712585ae635b”,“releaseId”:1994829}}
Jan 06 15:43:03 9db5f18 balenad[3088]: time=“2022-01-06T15:43:03.460527456Z” level=error msg=“stream copy error: reading from a closed fifo”
Jan 06 15:43:06 9db5f18 balenad[3088]: time=“2022-01-06T15:43:06.311130360Z” level=error msg=“00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container”
Jan 06 15:43:06 9db5f18 balenad[3088]: time=“2022-01-06T15:43:06.318270289Z” level=error msg=“Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown”
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 2000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:08 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:43:13 9db5f18 5702a46ac234[3088]: [event] Event: Service start {“service”:{“appId”:1775425,“serviceId”:841874,“serviceName”:“influxdb”,“commit”:“024a5f2e64c0c4fdfa4a712585ae635b”,“releaseId”:1994829}}
Jan 06 15:43:17 9db5f18 balenad[3088]: time=“2022-01-06T15:43:17.118199842Z” level=error msg=“stream copy error: reading from a closed fifo”
Jan 06 15:43:19 9db5f18 balenad[3088]: time=“2022-01-06T15:43:19.954881883Z” level=error msg=“00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container”
Jan 06 15:43:19 9db5f18 balenad[3088]: time=“2022-01-06T15:43:19.960671825Z” level=error msg=“Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown”
Jan 06 15:43:19 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 4000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks ()
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks ()
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:24 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:43:28 9db5f18 5702a46ac234[3088]: [event] Event: Service start {“service”:{“appId”:1775425,“serviceId”:841874,“serviceName”:“influxdb”,“commit”:“024a5f2e64c0c4fdfa4a712585ae635b”,“releaseId”:1994829}}
Jan 06 15:43:32 9db5f18 balenad[3088]: time=“2022-01-06T15:43:32.300883196Z” level=error msg=“stream copy error: reading from a closed fifo”
Jan 06 15:43:35 9db5f18 balenad[3088]: time=“2022-01-06T15:43:35.220046424Z” level=error msg=“00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container”
Jan 06 15:43:35 9db5f18 balenad[3088]: time=“2022-01-06T15:43:35.227147354Z” level=error msg=“Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown”
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 8000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks ()
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks ()
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:43 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:43:47 9db5f18 5702a46ac234[3088]: [event] Event: Service start {“service”:{“appId”:1775425,“serviceId”:841874,“serviceName”:“influxdb”,“commit”:“024a5f2e64c0c4fdfa4a712585ae635b”,“releaseId”:1994829}}
Jan 06 15:43:50 9db5f18 balenad[3088]: time=“2022-01-06T15:43:50.452489288Z” level=error msg=“stream copy error: reading from a closed fifo”
Jan 06 15:43:53 9db5f18 balenad[3088]: time=“2022-01-06T15:43:53.098094212Z” level=error msg=“00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container”
Jan 06 15:43:53 9db5f18 balenad[3088]: time=“2022-01-06T15:43:53.106892125Z” level=error msg=“Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown”
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 16000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks ()
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:[“start”]
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks ()
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:44:09 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:44:14 9db5f18 5702a46ac234[3088]: [event] Event: Service start {“service”:{“appId”:1775425,“serviceId”:841874,“serviceName”:“influxdb”,“commit”:“024a5f2e64c0c4fdfa4a712585ae635b”,“releaseId”:1994829}}
The following logs were captured immediately after trying the fix linked above:
Jan 06 15:42:24 9db5f18 5702a46ac234[3088]: [info] Supervisor v12.11.22 starting up...
Jan 06 15:42:28 9db5f18 5702a46ac234[3088]: [info] Setting host to discoverable
Jan 06 15:42:28 9db5f18 5702a46ac234[3088]: [warn] Invalid firewall mode: . Reverting to state: off
Jan 06 15:42:28 9db5f18 5702a46ac234[3088]: [info] Applying firewall mode: off
Jan 06 15:42:29 9db5f18 5702a46ac234[3088]: [debug] Starting systemd unit: avahi-daemon.service
Jan 06 15:42:29 9db5f18 5702a46ac234[3088]: [debug] Starting systemd unit: avahi-daemon.socket
Jan 06 15:42:29 9db5f18 5702a46ac234[3088]: [debug] Starting logging infrastructure
Jan 06 15:42:30 9db5f18 5702a46ac234[3088]: [info] Starting firewall
Jan 06 15:42:30 9db5f18 5702a46ac234[3088]: [debug] Performing database cleanup for container log timestamps
Jan 06 15:42:31 9db5f18 balenad[2740]: Parent process exited
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [info] Previous engine snapshot was not stored. Skipping cleanup.
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [debug] Handling of local mode switch is completed
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [success] Firewall mode applied
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [debug] Starting api binder
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [info] API Binder bound to: https://api.balena-cloud.com/v6/
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [event] Event: Supervisor start {}
Jan 06 15:42:34 9db5f18 5702a46ac234[3088]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [debug] Connectivity check enabled: true
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [debug] Starting periodic check for IP addresses
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [info] Reporting initial state, supervisor version and API info
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [debug] Skipping preloading
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [info] Starting API server
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] Supervisor API successfully started on port 48484
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [debug] VPN status path exists.
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [debug] Ensuring device is provisioned
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] VPN connection is active.
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] Waiting for connectivity...
Jan 06 15:42:37 9db5f18 5702a46ac234[3088]: [debug] Starting current state report
Jan 06 15:42:37 9db5f18 5702a46ac234[3088]: [debug] Starting target state poll
Jan 06 15:42:37 9db5f18 5702a46ac234[3088]: [debug] Spawning journald with: chroot /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
Jan 06 15:42:45 9db5f18 5702a46ac234[3088]: [event] Event: Service start {"service":{"appId":1775425,"serviceId":841874,"serviceName":"influxdb","commit":"024a5f2e64c0c4fdfa4a712585ae635b","releaseId":1994829}}
Jan 06 15:42:46 9db5f18 5702a46ac234[3088]: [info] Internet Connectivity: OK
Jan 06 15:42:49 9db5f18 balenad[3088]: time="2022-01-06T15:42:49.416886875Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 06 15:42:52 9db5f18 balenad[3088]: time="2022-01-06T15:42:52.181595625Z" level=error msg="00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container"
Jan 06 15:42:52 9db5f18 balenad[3088]: time="2022-01-06T15:42:52.188806554Z" level=error msg="Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown"
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] Updating failed, but there's another update scheduled immediately: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:42:58 9db5f18 5702a46ac234[3088]: [event] Event: Service start {"service":{"appId":1775425,"serviceId":841874,"serviceName":"influxdb","commit":"024a5f2e64c0c4fdfa4a712585ae635b","releaseId":1994829}}
Jan 06 15:43:03 9db5f18 balenad[3088]: time="2022-01-06T15:43:03.460527456Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 06 15:43:06 9db5f18 balenad[3088]: time="2022-01-06T15:43:06.311130360Z" level=error msg="00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container"
Jan 06 15:43:06 9db5f18 balenad[3088]: time="2022-01-06T15:43:06.318270289Z" level=error msg="Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown"
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 2000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:08 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:43:13 9db5f18 5702a46ac234[3088]: [event] Event: Service start {"service":{"appId":1775425,"serviceId":841874,"serviceName":"influxdb","commit":"024a5f2e64c0c4fdfa4a712585ae635b","releaseId":1994829}}
Jan 06 15:43:17 9db5f18 balenad[3088]: time="2022-01-06T15:43:17.118199842Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 06 15:43:19 9db5f18 balenad[3088]: time="2022-01-06T15:43:19.954881883Z" level=error msg="00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container"
Jan 06 15:43:19 9db5f18 balenad[3088]: time="2022-01-06T15:43:19.960671825Z" level=error msg="Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown"
Jan 06 15:43:19 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 4000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks (<anonymous>)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks (<anonymous>)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:24 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:43:28 9db5f18 5702a46ac234[3088]: [event] Event: Service start {"service":{"appId":1775425,"serviceId":841874,"serviceName":"influxdb","commit":"024a5f2e64c0c4fdfa4a712585ae635b","releaseId":1994829}}
Jan 06 15:43:32 9db5f18 balenad[3088]: time="2022-01-06T15:43:32.300883196Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 06 15:43:35 9db5f18 balenad[3088]: time="2022-01-06T15:43:35.220046424Z" level=error msg="00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container"
Jan 06 15:43:35 9db5f18 balenad[3088]: time="2022-01-06T15:43:35.227147354Z" level=error msg="Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown"
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 8000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks (<anonymous>)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks (<anonymous>)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:43 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:43:47 9db5f18 5702a46ac234[3088]: [event] Event: Service start {"service":{"appId":1775425,"serviceId":841874,"serviceName":"influxdb","commit":"024a5f2e64c0c4fdfa4a712585ae635b","releaseId":1994829}}
Jan 06 15:43:50 9db5f18 balenad[3088]: time="2022-01-06T15:43:50.452489288Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 06 15:43:53 9db5f18 balenad[3088]: time="2022-01-06T15:43:53.098094212Z" level=error msg="00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container"
Jan 06 15:43:53 9db5f18 balenad[3088]: time="2022-01-06T15:43:53.106892125Z" level=error msg="Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown"
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 16000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks (<anonymous>)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks (<anonymous>)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:44:09 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:44:14 9db5f18 5702a46ac234[3088]: [event] Event: Service start {"service":{"appId":1775425,"serviceId":841874,"serviceName":"influxdb","commit":"024a5f2e64c0c4fdfa4a712585ae635b","releaseId":1994829}}
The following logs were captured after trying the fix linked above:
Jan 06 15:42:24 9db5f18 5702a46ac234[3088]: [info] Supervisor v12.11.22 starting up...
Jan 06 15:42:28 9db5f18 5702a46ac234[3088]: [info] Setting host to discoverable
Jan 06 15:42:28 9db5f18 5702a46ac234[3088]: [warn] Invalid firewall mode: . Reverting to state: off
Jan 06 15:42:28 9db5f18 5702a46ac234[3088]: [info] Applying firewall mode: off
Jan 06 15:42:29 9db5f18 5702a46ac234[3088]: [debug] Starting systemd unit: avahi-daemon.service
Jan 06 15:42:29 9db5f18 5702a46ac234[3088]: [debug] Starting systemd unit: avahi-daemon.socket
Jan 06 15:42:29 9db5f18 5702a46ac234[3088]: [debug] Starting logging infrastructure
Jan 06 15:42:30 9db5f18 5702a46ac234[3088]: [info] Starting firewall
Jan 06 15:42:30 9db5f18 5702a46ac234[3088]: [debug] Performing database cleanup for container log timestamps
Jan 06 15:42:31 9db5f18 balenad[2740]: Parent process exited
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [info] Previous engine snapshot was not stored. Skipping cleanup.
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [debug] Handling of local mode switch is completed
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [success] Firewall mode applied
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [debug] Starting api binder
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [info] API Binder bound to: https://api.balena-cloud.com/v6/
Jan 06 15:42:33 9db5f18 5702a46ac234[3088]: [event] Event: Supervisor start {}
Jan 06 15:42:34 9db5f18 5702a46ac234[3088]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [debug] Connectivity check enabled: true
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [debug] Starting periodic check for IP addresses
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [info] Reporting initial state, supervisor version and API info
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [debug] Skipping preloading
Jan 06 15:42:35 9db5f18 5702a46ac234[3088]: [info] Starting API server
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] Supervisor API successfully started on port 48484
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [debug] VPN status path exists.
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [debug] Ensuring device is provisioned
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] VPN connection is active.
Jan 06 15:42:36 9db5f18 5702a46ac234[3088]: [info] Waiting for connectivity...
Jan 06 15:42:37 9db5f18 5702a46ac234[3088]: [debug] Starting current state report
Jan 06 15:42:37 9db5f18 5702a46ac234[3088]: [debug] Starting target state poll
Jan 06 15:42:37 9db5f18 5702a46ac234[3088]: [debug] Spawning journald with: chroot /mnt/root journalctl -a --follow -o json _SYSTEMD_UNIT=balena.service
Jan 06 15:42:45 9db5f18 5702a46ac234[3088]: [event] Event: Service start {"service":{"appId":1775425,"serviceId":841874,"serviceName":"influxdb","commit":"024a5f2e64c0c4fdfa4a712585ae635b","releaseId":1994829}}
Jan 06 15:42:46 9db5f18 5702a46ac234[3088]: [info] Internet Connectivity: OK
Jan 06 15:42:49 9db5f18 balenad[3088]: time="2022-01-06T15:42:49.416886875Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 06 15:42:52 9db5f18 balenad[3088]: time="2022-01-06T15:42:52.181595625Z" level=error msg="00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container"
Jan 06 15:42:52 9db5f18 balenad[3088]: time="2022-01-06T15:42:52.188806554Z" level=error msg="Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown"
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] Updating failed, but there's another update scheduled immediately: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:42:52 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:42:58 9db5f18 5702a46ac234[3088]: [event] Event: Service start {"service":{"appId":1775425,"serviceId":841874,"serviceName":"influxdb","commit":"024a5f2e64c0c4fdfa4a712585ae635b","releaseId":1994829}}
Jan 06 15:43:03 9db5f18 balenad[3088]: time="2022-01-06T15:43:03.460527456Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 06 15:43:06 9db5f18 balenad[3088]: time="2022-01-06T15:43:06.311130360Z" level=error msg="00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container"
Jan 06 15:43:06 9db5f18 balenad[3088]: time="2022-01-06T15:43:06.318270289Z" level=error msg="Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown"
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 2000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:06 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:08 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:43:13 9db5f18 5702a46ac234[3088]: [event] Event: Service start {"service":{"appId":1775425,"serviceId":841874,"serviceName":"influxdb","commit":"024a5f2e64c0c4fdfa4a712585ae635b","releaseId":1994829}}
Jan 06 15:43:17 9db5f18 balenad[3088]: time="2022-01-06T15:43:17.118199842Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 06 15:43:19 9db5f18 balenad[3088]: time="2022-01-06T15:43:19.954881883Z" level=error msg="00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container"
Jan 06 15:43:19 9db5f18 balenad[3088]: time="2022-01-06T15:43:19.960671825Z" level=error msg="Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown"
Jan 06 15:43:19 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 4000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks (<anonymous>)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks (<anonymous>)
Jan 06 15:43:20 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:24 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:43:28 9db5f18 5702a46ac234[3088]: [event] Event: Service start {"service":{"appId":1775425,"serviceId":841874,"serviceName":"influxdb","commit":"024a5f2e64c0c4fdfa4a712585ae635b","releaseId":1994829}}
Jan 06 15:43:32 9db5f18 balenad[3088]: time="2022-01-06T15:43:32.300883196Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 06 15:43:35 9db5f18 balenad[3088]: time="2022-01-06T15:43:35.220046424Z" level=error msg="00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container"
Jan 06 15:43:35 9db5f18 balenad[3088]: time="2022-01-06T15:43:35.227147354Z" level=error msg="Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown"
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 8000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks (<anonymous>)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks (<anonymous>)
Jan 06 15:43:35 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:43 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:43:47 9db5f18 5702a46ac234[3088]: [event] Event: Service start {"service":{"appId":1775425,"serviceId":841874,"serviceName":"influxdb","commit":"024a5f2e64c0c4fdfa4a712585ae635b","releaseId":1994829}}
Jan 06 15:43:50 9db5f18 balenad[3088]: time="2022-01-06T15:43:50.452489288Z" level=error msg="stream copy error: reading from a closed fifo"
Jan 06 15:43:53 9db5f18 balenad[3088]: time="2022-01-06T15:43:53.098094212Z" level=error msg="00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca cleanup: failed to delete container from containerd: no such container"
Jan 06 15:43:53 9db5f18 balenad[3088]: time="2022-01-06T15:43:53.106892125Z" level=error msg="Handler for POST /containers/00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca/start returned error: task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown"
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] Scheduling another update attempt in 16000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks (<anonymous>)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - task 00ac36361ea288c0aca1b8f889f6fa2ad1a5d724e5a103056c6c1ffae09505ca already exists: unknown Steps:["start"]
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at fn (/usr/src/app/dist/app.js:6:8690)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at runMicrotasks (<anonymous>)
Jan 06 15:43:53 9db5f18 5702a46ac234[3088]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Jan 06 15:44:09 9db5f18 5702a46ac234[3088]: [info] Applying target state
Jan 06 15:44:14 9db5f18 5702a46ac234[3088]: [event] Event: Service start {"service":{"appId":1775425,"serviceId":841874,"serviceName":"influxdb","commit":"024a5f2e64c0c4fdfa4a712585ae635b","releaseId":1994829}}
I’ve tried to upload the logs but they’re being blocked by the spam filter.
After repeated restarts I was able to get the services running again. However after running for a few hours they stopped again without writing anything to the log.
Attempted to make changes to the a few of the images to capture more information - however the update stalled at 5% and cannot be made to continue.
Hello @disparlure first of all apologizes for the long delay!
Are you following the conversation there was on the github issue my colleague suggested? Port already in use, because proxy keeps binding to the wrong container IP · Issue #272 · balena-os/balena-engine · GitHub
Actually i associated your message with the issue, and if it gets solved we will let you know. BTW you are welcome to contribute on our github repositories.