Hi,
i’ve a problem with our nuc’s after rebbot or a power failure.
The 5 containers has a loop with Service exited and Service starting, Service killed …
My Workaround is, deploying a older release and then again the current release.
I don’t know where the problem is. Maybe you can help me
Thx
hi, are there any interesting logs from the supervisor?
Hi,
oh very good input…you mean journalctl -f
I’ve an smb container which comes not up:
Bind for 0.0.0.0:445 failed: port is already allocated
Ok, maybe there is a problem, but why? I’ve no other container which is using port 445 at 0.0.0.0
And i’ve an nginx proxy container…but he says nothing
Sep 08 11:57:17 ad8a30c resin-supervisor[26559]: [api] POST /v2/applications/1676803/restart-service 404 - 19.482 ms
The smb service says “Installed”
The Proxy sarvice says “Downloaded”
Maybe, could it a problem by using something like this:
proxy:
image: *****/nuc-proxy:1.0.1
restart: unless-stopped
networks:
sat_net:
ipv4_address: 172.50.0.2
ports:
- “80:80”
environment:
- NGINX_PORT=80
networks:
sat_net:
ipam:
driver: default
config:
- subnet: 172.50.0.0/16
For each container i fixed the ip address.
Hi Philpp, my team-mates was suggesting pulling supervisor logs from the hostOS (you can access it from the web terminal in your dashboard)
systemctl status resin-supervisor -l
journalctl -a | grep supervisor
Sep 08 12:48:28 ad8a30c NetworkManager[1012]: [1599569308.3719] device (supervisor0): carrier: link connected
Sep 08 12:48:28 ad8a30c resin-supervisor[26559]: [event] Event: Service started {“service”:{“appId”:1676803,“serviceId”:568445,“serviceName”:“controller”,“releaseId”:1512078}}
Sep 08 12:48:31 ad8a30c resin-supervisor[26559]: [event] Event: Service started {“service”:{“appId”:1676803,“serviceId”:568446,“serviceName”:“wildfly”,“releaseId”:1512078}}
Sep 08 12:51:11 ad8a30c resin-supervisor[26559]: [debug] Attempting container log timestamp flush…
Sep 08 12:51:11 ad8a30c resin-supervisor[26559]: [debug] Container log timestamp flush complete
Sep 08 12:51:26 ad8a30c resin-supervisor[26559]: [api] GET /v1/healthy 200 - 1.601 ms
Sep 08 12:56:26 ad8a30c resin-supervisor[26559]: [api] GET /v1/healthy 200 - 1.436 ms
Sep 08 13:01:11 ad8a30c resin-supervisor[26559]: [debug] Attempting container log timestamp flush…
Sep 08 13:01:11 ad8a30c resin-supervisor[26559]: [debug] Container log timestamp flush complete
Sep 08 13:01:26 ad8a30c resin-supervisor[26559]: [api] GET /v1/healthy 200 - 1.627 ms
Sep 08 13:03:27 ad8a30c resin-supervisor[26559]: [info] Applying target state
Sep 08 13:03:27 ad8a30c resin-supervisor[26559]: [event] Event: Service start {“service”:{“appId”:1676803,“serviceId”:568447,“serviceName”:“exchange”,“releaseId”:1512078}}
Sep 08 13:03:27 ad8a30c resin-supervisor[26559]: [event] Event: Service kill {“service”:{“appId”:1676803,“serviceId”:568446,“serviceName”:“wildfly”,“releaseId”:1512078}}
Sep 08 13:03:27 ad8a30c resin-supervisor[26559]: [event] Event: Service kill {“service”:{“appId”:1676803,“serviceId”:568445,“serviceName”:“controller”,“releaseId”:1512078}}
Sep 08 13:03:27 ad8a30c resin-supervisor[26559]: [error] Scheduling another update attempt in 900000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - driver failed programming external connectivity on endpoint exchange_2659794_1512078 (dd262a38193ac269a8a9e20908f612d13373682b7383890c23c3ed88cc61ef28): Bind for 0.0.0.0:445 failed: port is already allocated Steps:[“start”,“start”,“kill”,“kill”]
Sep 08 13:03:27 ad8a30c resin-supervisor[26559]: [error] at /usr/src/app/dist/app.js:2:2958395
Sep 08 13:03:27 ad8a30c resin-supervisor[26559]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - driver failed programming external connectivity on endpoint exchange_2659794_1512078 (dd262a38193ac269a8a9e20908f612d13373682b7383890c23c3ed88cc61ef28): Bind for 0.0.0.0:445 failed: port is already allocated Steps:[“start”,“start”,“kill”,“kill”]
Sep 08 13:03:27 ad8a30c resin-supervisor[26559]: [error] at /usr/src/app/dist/app.js:2:2958395
Sep 08 13:03:27 ad8a30c resin-supervisor[26559]: [event] Event: Service install {“service”:{“appId”:1676803,“serviceId”:568449,“serviceName”:“proxy”,“releaseId”:1512078}}
Sep 08 13:03:27 ad8a30c resin-supervisor[26559]: [event] Event: Service installed {“service”:{“appId”:1676803,“serviceId”:568449,“serviceName”:“proxy”,“releaseId”:1512078}}
Sep 08 13:03:27 ad8a30c resin-supervisor[26559]: [event] Event: Service start {“service”:{“appId”:1676803,“serviceId”:568449,“serviceName”:“proxy”,“releaseId”:1512078}}
Sep 08 13:03:28 ad8a30c resin-supervisor[26559]: [event] Event: Service started {“service”:{“appId”:1676803,“serviceId”:568449,“serviceName”:“proxy”,“releaseId”:1512078}}
Sep 08 13:03:38 ad8a30c kernel: supervisor0: port 1(vethc785109) entered disabled state
Sep 08 13:03:38 ad8a30c kernel: supervisor0: port 1(vethc785109) entered disabled state
Sep 08 13:03:38 ad8a30c kernel: supervisor0: port 1(vethc785109) entered disabled state
Sep 08 13:03:38 ad8a30c NetworkManager[1012]: [1599570218.2975] device (vethc785109): released from master device supervisor0
Sep 08 13:03:38 ad8a30c resin-supervisor[26559]: [event] Event: Service exit {“service”:{“appId”:1676803,“serviceId”:568445,“serviceName”:“controller”,“releaseId”:1512078}}
Sep 08 13:03:38 ad8a30c resin-supervisor[26559]: [event] Event: Service stop {“service”:{“appId”:1676803,“serviceId”:568445,“serviceName”:“controller”,“releaseId”:1512078}}
Sep 08 13:03:39 ad8a30c resin-supervisor[26559]: [event] Event: Service exit {“service”:{“appId”:1676803,“serviceId”:568446,“serviceName”:“wildfly”,“releaseId”:1512078}}
Sep 08 13:03:39 ad8a30c resin-supervisor[26559]: [event] Event: Service stop {“service”:{“appId”:1676803,“serviceId”:568446,“serviceName”:“wildfly”,“releaseId”:1512078}}
Sep 08 13:06:27 ad8a30c resin-supervisor[26559]: [api] GET /v1/healthy 200 - 1.614 ms
Sep 08 13:11:11 ad8a30c resin-supervisor[26559]: [debug] Attempting container log timestamp flush…
Sep 08 13:11:11 ad8a30c resin-supervisor[26559]: [debug] Container log timestamp flush complete
Sep 08 13:11:27 ad8a30c resin-supervisor[26559]: [api] GET /v1/healthy 200 - 1.368 ms
Sep 08 13:16:27 ad8a30c resin-supervisor[26559]: [api] GET /v1/healthy 200 - 1.828 ms
Sep 08 13:18:27 ad8a30c resin-supervisor[26559]: [info] Applying target state
Sep 08 13:18:27 ad8a30c resin-supervisor[26559]: [event] Event: Service start {“service”:{“appId”:1676803,“serviceId”:568447,“serviceName”:“exchange”,“releaseId”:1512078}}
Sep 08 13:18:27 ad8a30c resin-supervisor[26559]: [event] Event: Service kill {“service”:{“appId”:1676803,“serviceId”:568449,“serviceName”:“proxy”,“releaseId”:1512078}}
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [error] Scheduling another update attempt in 900000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - driver failed programming external connectivity on endpoint exchange_2659794_1512078 (ea86de8c7f3368769d3456529b7eb69c4d7d7b876c9351f34bbd9e5a8aa00e8a): Bind for 0.0.0.0:445 failed: port is already allocated Steps:[“start”,“start”,“start”,“kill”]
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [error] at /usr/src/app/dist/app.js:2:2958395
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - driver failed programming external connectivity on endpoint exchange_2659794_1512078 (ea86de8c7f3368769d3456529b7eb69c4d7d7b876c9351f34bbd9e5a8aa00e8a): Bind for 0.0.0.0:445 failed: port is already allocated Steps:[“start”,“start”,“start”,“kill”]
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [error] at /usr/src/app/dist/app.js:2:2958395
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [event] Event: Service install {“service”:{“appId”:1676803,“serviceId”:568445,“serviceName”:“controller”,“releaseId”:1512078}}
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [event] Event: Service install {“service”:{“appId”:1676803,“serviceId”:568446,“serviceName”:“wildfly”,“releaseId”:1512078}}
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [event] Event: Service installed {“service”:{“appId”:1676803,“serviceId”:568446,“serviceName”:“wildfly”,“releaseId”:1512078}}
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [event] Event: Service start {“service”:{“appId”:1676803,“serviceId”:568446,“serviceName”:“wildfly”,“releaseId”:1512078}}
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [event] Event: Service installed {“service”:{“appId”:1676803,“serviceId”:568445,“serviceName”:“controller”,“releaseId”:1512078}}
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [event] Event: Service start {“service”:{“appId”:1676803,“serviceId”:568445,“serviceName”:“controller”,“releaseId”:1512078}}
Sep 08 13:18:28 ad8a30c kernel: supervisor0: port 1(vethf1a0ece) entered blocking state
Sep 08 13:18:28 ad8a30c kernel: supervisor0: port 1(vethf1a0ece) entered disabled state
Sep 08 13:18:28 ad8a30c kernel: supervisor0: port 1(vethf1a0ece) entered blocking state
Sep 08 13:18:28 ad8a30c kernel: supervisor0: port 1(vethf1a0ece) entered forwarding state
Sep 08 13:18:28 ad8a30c kernel: supervisor0: port 1(vethf1a0ece) entered disabled state
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [event] Event: Service exit {“service”:{“appId”:1676803,“serviceId”:568449,“serviceName”:“proxy”,“releaseId”:1512078}}
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [event] Event: Service stop {“service”:{“appId”:1676803,“serviceId”:568449,“serviceName”:“proxy”,“releaseId”:1512078}}
Sep 08 13:18:28 ad8a30c kernel: supervisor0: port 1(vethf1a0ece) entered blocking state
Sep 08 13:18:28 ad8a30c kernel: supervisor0: port 1(vethf1a0ece) entered forwarding state
Sep 08 13:18:28 ad8a30c NetworkManager[1012]: [1599571108.6933] device (supervisor0): carrier: link connected
Sep 08 13:18:28 ad8a30c resin-supervisor[26559]: [event] Event: Service started {“service”:{“appId”:1676803,“serviceId”:568445,“serviceName”:“controller”,“releaseId”:1512078}}
Sep 08 13:18:31 ad8a30c resin-supervisor[26559]: [event] Event: Service started {“service”:{“appId”:1676803,“serviceId”:568446,“serviceName”:“wildfly”,“releaseId”:1512078}}
Sep 08 13:21:11 ad8a30c resin-supervisor[26559]: [debug] Attempting container log timestamp flush…
Sep 08 13:21:11 ad8a30c resin-supervisor[26559]: [debug] Container log timestamp flush complete
Sep 08 13:21:27 ad8a30c resin-supervisor[26559]: [api] GET /v1/healthy 200 - 1.804 ms
Sep 08 13:26:28 ad8a30c resin-supervisor[26559]: [api] GET /v1/healthy 200 - 5.205 ms
Sep 08 13:31:11 ad8a30c resin-supervisor[26559]: [debug] Attempting container log timestamp flush…
Sep 08 13:31:11 ad8a30c resin-supervisor[26559]: [debug] Container log timestamp flush complete
Sep 08 13:31:28 ad8a30c resin-supervisor[26559]: [api] GET /v1/healthy 200 - 1.661 ms
Sep 08 13:33:28 ad8a30c resin-supervisor[26559]: [info] Applying target state
Sep 08 13:33:28 ad8a30c resin-supervisor[26559]: [event] Event: Service start {“service”:{“appId”:1676803,“serviceId”:568447,“serviceName”:“exchange”,“releaseId”:1512078}}
Sep 08 13:33:28 ad8a30c resin-supervisor[26559]: [event] Event: Service kill {“service”:{“appId”:1676803,“serviceId”:568446,“serviceName”:“wildfly”,“releaseId”:1512078}}
Sep 08 13:33:28 ad8a30c resin-supervisor[26559]: [event] Event: Service kill {“service”:{“appId”:1676803,“serviceId”:568445,“serviceName”:“controller”,“releaseId”:1512078}}
Sep 08 13:33:28 ad8a30c resin-supervisor[26559]: [error] Scheduling another update attempt in 900000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 500) server error - driver failed programming external connectivity on endpoint exchange_2659794_1512078 (ab2c89f6a96f05e635d62106c81122f26a73c56922ac4ed53a70707eb19c54b9): Bind for 0.0.0.0:445 failed: port is already allocated Steps:[“start”,“start”,“kill”,“kill”]
Sep 08 13:33:28 ad8a30c resin-supervisor[26559]: [error] at /usr/src/app/dist/app.js:2:2958395
Sep 08 13:33:28 ad8a30c resin-supervisor[26559]: [event] Event: Service install {“service”:{“appId”:1676803,“serviceId”:568449,“serviceName”:“proxy”,“releaseId”:1512078}}
Sep 08 13:33:28 ad8a30c resin-supervisor[26559]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 500) server error - driver failed programming external connectivity on endpoint exchange_2659794_1512078 (ab2c89f6a96f05e635d62106c81122f26a73c56922ac4ed53a70707eb19c54b9): Bind for 0.0.0.0:445 failed: port is already allocated Steps:[“start”,“start”,“kill”,“kill”]
Sep 08 13:33:28 ad8a30c resin-supervisor[26559]: [error] at /usr/src/app/dist/app.js:2:2958395
Sep 08 13:33:28 ad8a30c resin-supervisor[26559]: [event] Event: Service installed {“service”:{“appId”:1676803,“serviceId”:568449,“serviceName”:“proxy”,“releaseId”:1512078}}
Sep 08 13:33:28 ad8a30c resin-supervisor[26559]: [event] Event: Service start {“service”:{“appId”:1676803,“serviceId”:568449,“serviceName”:“proxy”,“releaseId”:1512078}}
Sep 08 13:33:28 ad8a30c resin-supervisor[26559]: [event] Event: Service started {“service”:{“appId”:1676803,“serviceId”:568449,“serviceName”:“proxy”,“releaseId”:1512078}}
Sep 08 13:33:38 ad8a30c kernel: supervisor0: port 1(vethf1a0ece) entered disabled state
Sep 08 13:33:38 ad8a30c kernel: supervisor0: port 1(vethf1a0ece) entered disabled state
Sep 08 13:33:38 ad8a30c kernel: supervisor0: port 1(vethf1a0ece) entered disabled state
Sep 08 13:33:38 ad8a30c NetworkManager[1012]: [1599572018.6118] device (vethf1a0ece): released from master device supervisor0
Sep 08 13:33:38 ad8a30c resin-supervisor[26559]: [event] Event: Service exit {“service”:{“appId”:1676803,“serviceId”:568445,“serviceName”:“controller”,“releaseId”:1512078}}
Sep 08 13:33:38 ad8a30c resin-supervisor[26559]: [event] Event: Service stop {“service”:{“appId”:1676803,“serviceId”:568445,“serviceName”:“controller”,“releaseId”:1512078}}
Sep 08 13:33:39 ad8a30c resin-supervisor[26559]: [event] Event: Service exit {“service”:{“appId”:1676803,“serviceId”:568446,“serviceName”:“wildfly”,“releaseId”:1512078}}
Sep 08 13:33:39 ad8a30c resin-supervisor[26559]: [event] Event: Service stop {“service”:{“appId”:1676803,“serviceId”:568446,“serviceName”:“wildfly”,“releaseId”:1512078}}
Sep 08 13:36:28 ad8a30c resin-supervisor[26559]: [api] GET /v1/healthy 200 - 1.535 ms
Sep 08 13:41:11 ad8a30c resin-supervisor[26559]: [debug] Attempting container log timestamp flush…
Sep 08 13:41:11 ad8a30c resin-supervisor[26559]: [debug] Container log timestamp flush complete
Sep 08 13:41:28 ad8a30c resin-supervisor[26559]: [api] GET /v1/healthy 200 - 1.382 ms
thanks for the logs, I pinged the supervisor maintainer to give a look at them and advice on the next debugging steps
perfect…thx a lot
This looks likely because the port 445 is already in use, and so the container cannot be started.
Error: Failed to apply state transition steps. (HTTP code 500) server error - driver failed programming external connectivity on endpoint exchange_2659794_1512078 (dd262a38193ac269a8a9e20908f612d13373682b7383890c23c3ed88cc61ef28): Bind for 0.0.0.0:445 failed: port is already allocated
If you could get your device into the bad state, then on the host run balena ps and see what containers are running etc. Either you have another app using 445, or a container created by the Supervisor is not being linked to the application for some reason.
To clarify, the Supervisor will not be able to start a container if the port it in use. So it will drop into a start/kill loop until it manages to start.
Nope no running container with port 445 allocated…
The interesting is, when i made an update, there is no problem.
the problem is still present, when i made a reboot or power off the device.
We have the use case, we flash the nuc to the application and made some configs.
After that, we turn of the deive and the technican goes to the medical center.
In the most cases the nuc comes online and is accessible, but he has a loop at the deployment.
The only help at this point is the move the device short to another application or made a release change.
Sometime a reboot helps to.
Updates to a new release is also never a problem.
If check and check again because of the port 445, but if no expose to 445 from a container.
i have cups running in host mode, could cups maybe use 445 too?
Or i have a zeroconf container running on host mode. the rest of the containers are running not in host mode.
The main container i use for 445 is the smb server…this container expose 445
How can i find out?
Hi there,
If you run netstat -tulpn you should be able to see what processes are listening to ports on your machine. Try sudo netstat -tulpn if you can’t see anything on the port without sudo. You should be able to get a process ID and investigate from there
root@7c2214f:~# journalctl -a | grep supervisor | grep error
Sep 15 11:19:25 7c2214f resin-supervisor[5436]: [error] Failed to get target state for device: InternalInconsistencyError: No delay specified in scheduledApply
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [event] Event: Network removal error {“error”:{“message”:"(HTTP code 403) unexpected - error while removing network: network 1676803_sat_net id 05015e1676547cb04924f4ce15928cdb3a81aeed3b40c7a2d6e8fe886abcd073 has active endpoints “,“stack”:“Error: (HTTP code 403) unexpected - error while removing network: network 1676803_sat_net id 05015e1676547cb04924f4ce15928cdb3a81aeed3b40c7a2d6e8fe886abcd073 has active endpoints \n at /usr/src/app/dist/app.js:14:1391724\n at /usr/src/app/dist/app.js:14:1391656\n at Modem.buildPayload (/usr/src/app/dist/app.js:14:1391676)\n at IncomingMessage. (/usr/src/app/dist/app.js:14:1390929)\n at IncomingMessage.emit (events.js:322:22)\n at endReadableNT (_stream_readable.js:1187:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)”}}
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] Updating failed, but there’s another update scheduled immediately: Error: Failed to apply state transition steps. (HTTP code 403) unexpected - error while removing network: network 1676803_sat_net id 05015e1676547cb04924f4ce15928cdb3a81aeed3b40c7a2d6e8fe886abcd073 has active endpoints Steps:[“start”,“start”,“start”,“start”,“start”,“removeNetwork”]
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] at /usr/src/app/dist/app.js:2:2958395
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] at runMicrotasks ()
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 403) unexpected - error while removing network: network 1676803_sat_net id 05015e1676547cb04924f4ce15928cdb3a81aeed3b40c7a2d6e8fe886abcd073 has active endpoints Steps:[“start”,“start”,“start”,“start”,“start”,“removeNetwork”]
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] at /usr/src/app/dist/app.js:2:2958395
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] at runMicrotasks ()
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [event] Event: Network removal error {“error”:{“message”:”(HTTP code 403) unexpected - error while removing network: network 1676803_sat_net id 05015e1676547cb04924f4ce15928cdb3a81aeed3b40c7a2d6e8fe886abcd073 has active endpoints “,“stack”:“Error: (HTTP code 403) unexpected - error while removing network: network 1676803_sat_net id 05015e1676547cb04924f4ce15928cdb3a81aeed3b40c7a2d6e8fe886abcd073 has active endpoints \n at /usr/src/app/dist/app.js:14:1391724\n at /usr/src/app/dist/app.js:14:1391656\n at Modem.buildPayload (/usr/src/app/dist/app.js:14:1391676)\n at IncomingMessage. (/usr/src/app/dist/app.js:14:1390929)\n at IncomingMessage.emit (events.js:322:22)\n at endReadableNT (_stream_readable.js:1187:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)”}}
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] Scheduling another update attempt in 2000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 403) unexpected - error while removing network: network 1676803_sat_net id 05015e1676547cb04924f4ce15928cdb3a81aeed3b40c7a2d6e8fe886abcd073 has active endpoints Steps:[“start”,“start”,“start”,“start”,“removeNetwork”]
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] at /usr/src/app/dist/app.js:2:2958395
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] at runMicrotasks ()
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 403) unexpected - error while removing network: network 1676803_sat_net id 05015e1676547cb04924f4ce15928cdb3a81aeed3b40c7a2d6e8fe886abcd073 has active endpoints Steps:[“start”,“start”,“start”,“start”,“removeNetwork”]
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] at /usr/src/app/dist/app.js:2:2958395
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] at runMicrotasks ()
Sep 15 11:29:50 7c2214f resin-supervisor[11279]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Sep 15 11:30:12 7c2214f resin-supervisor[11279]: [event] Event: Network removal error {“error”:{“message”:”(HTTP code 403) unexpected - error while removing network: network 1676803_sat_net id 05015e1676547cb04924f4ce15928cdb3a81aeed3b40c7a2d6e8fe886abcd073 has active endpoints ",“stack”:“Error: (HTTP code 403) unexpected - error while removing network: network 1676803_sat_net id 05015e1676547cb04924f4ce15928cdb3a81aeed3b40c7a2d6e8fe886abcd073 has active endpoints \n at /usr/src/app/dist/app.js:14:1391724\n at /usr/src/app/dist/app.js:14:1391656\n at Modem.buildPayload (/usr/src/app/dist/app.js:14:1391676)\n at IncomingMessage. (/usr/src/app/dist/app.js:14:1390929)\n at IncomingMessage.emit (events.js:322:22)\n at endReadableNT (_stream_readable.js:1187:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)”}}
Sep 15 11:30:12 7c2214f resin-supervisor[11279]: [error] Scheduling another update attempt in 4000ms due to failure: Error: Failed to apply state transition steps. (HTTP code 403) unexpected - error while removing network: network 1676803_sat_net id 05015e1676547cb04924f4ce15928cdb3a81aeed3b40c7a2d6e8fe886abcd073 has active endpoints Steps:[“start”,“start”,“start”,“start”,“removeNetwork”]
Sep 15 11:30:12 7c2214f resin-supervisor[11279]: [error] at /usr/src/app/dist/app.js:2:2958395
Sep 15 11:30:12 7c2214f resin-supervisor[11279]: [error] at runMicrotasks ()
Sep 15 11:30:12 7c2214f resin-supervisor[11279]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Sep 15 11:30:12 7c2214f resin-supervisor[11279]: [error] Device state apply error Error: Failed to apply state transition steps. (HTTP code 403) unexpected - error while removing network: network 1676803_sat_net id 05015e1676547cb04924f4ce15928cdb3a81aeed3b40c7a2d6e8fe886abcd073 has active endpoints Steps:[“start”,“start”,“start”,“start”,“removeNetwork”]
Sep 15 11:30:12 7c2214f resin-supervisor[11279]: [error] at /usr/src/app/dist/app.js:2:2958395
Sep 15 11:30:12 7c2214f resin-supervisor[11279]: [error] at runMicrotasks ()
Sep 15 11:30:12 7c2214f resin-supervisor[11279]: [error] at processTicksAndRejections (internal/process/task_queues.js:97:5)
Ohhh maybe i find the problem…but i need your help
Balena has problems to remove the docker network, could this be?
Hi there – thanks for the additional information. Can I ask you please for two additional bits of information? At a time when your device is unhappy (when it’s stuck in the restart loop), can you please:
netstat -tan in the hostOS of your device, and paste the output here.This will help us get information we need about what might be holding open the port.
Thanks, and please let us know if you run into any problems.
Hugh
Hi again,
I’m curious to know if you’ve had a chance yet to follow my colleague’s suggestions, so we can better understand what might be causing that port to remain open?
Please let us know if and when you are able to run those commands!
Hi,
i’ve found a solution for me. The Problem was, i fixed the ip in the docker-compose file, because i had containers in host and also in a docker network running and i used a nginx proxy to make it accessible over port 80. the solution was to use two proxys. one in the sat network and then one in hostmode. so have no need to fix the ip range and ips of the containers. after this change, i had never a problem after this change
Thanks for the followup note, Philipp. Glad to hear you got it resolved!