I find that I spend more time than I’d like to waiting for balena cloud to update my device from one release to the next. Is there a way to force balena cloud to update/upgrade to the newest release?
Hi @KingstonSteele, this doesn’t sound as expected. Can you give us a bit more information about exactly what you are experiencing and anything about your setup that might be relevant (dev vs. prod image, disabled VPN etc…) Does the device have intermittent connectivity? The API will notify the Supervisor there is a new release available but if that was not received then the next time it would be picked up would be via the regular polling interval (which is configurable). How long are you typically waiting?
Right now its a super simple application with a python script, wifi-connect, and an MQTT broker. I’m using a production img, haven’t disabled or enabled anything. It’s really bare bones. The connection is great but I’ll wait maybe a half hour before it updates. It shows the current release and then target but won’t update to that target release for quite sometime.
Thanks for the update, to investigate further can you output some logs from the device Supervisor and paste them here, something like journalctl -n 200 --unit resin-supervisor. Also can you confirm the balenaOS version you are using and device type?
Hi,
Additionally, could you let us know the device and the version of the OS that you’re using on your device, so we can try and reproduce this issue using the same steps you are? Downloads should start almost immediately after they’re built and stored in the balena registry due to the way that the Supervisor on the device is signalled.
Best regards,
Heds
balenaOS 2.47.0+rev1
journalctl -n 200 --unit resin-supervisor
output:
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at c (/usr/src/app/dist/app.js:9:77523)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at d (/usr/src/app/dist/app.js:312:329886)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at p (/usr/src/app/dist/app.js:312:329825)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at s._drainQueues (/usr/src/app/dist/app.js:312:331345)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at Immediate.drainQueues [as _onImmediate] (/usr/src/app/dist/app.js:312:329567)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at runCallback (timers.js:705:18)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at tryOnImmediate (timers.js:676:5)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at processImmediate (timers.js:658:5)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] Device state apply error Error: Failed to apply state transition steps. Updates are locked: EEXIS>
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at /usr/src/app/dist/app.js:614:16375
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at c (/usr/src/app/dist/app.js:9:77523)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at O._settlePromiseFromHandler (/usr/src/app/dist/app.js:312:325441)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at O._settlePromise (/usr/src/app/dist/app.js:312:326241)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at O._settlePromise0 (/usr/src/app/dist/app.js:312:326940)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at O._settlePromises (/usr/src/app/dist/app.js:312:328181)
Feb 19 04:50:31 51f2138 resin-supervisor[1700]: [error] at d (/usr/src/app/dist/app.js:312:329886)
I just noticed that when I try to shut down I get this message/notification:
“The device was not shutdown because an update lock has been set. You can force the shutdown from the actions page.”
Hi,
According to the supervisor logs, looks like you have a lock that stops the device from updating, it’s a balena feature and you can find more details here https://www.balena.io/docs/learn/deploy/release-strategy/update-locking/#application-update-locks or https://www.balena.io/docs/learn/manage/actions/#update-locking.
Can you please check and remove the lock if you do not need it?
What would be the reason that that lock is happening? I toggled it off in the drop down on the dashboard but I’d like to figure out how it was locked to begin with. I sometimes flash using preload so it shows “factory build”. Perhaps this is the reason?
Hello,
Application update locks are set by user application code (not the balena system). The idea is to stop the balena supervisor from applying updates while the application is doing something critical. It would seem that some part of your application set this lock, and did not remove it later - perhaps the app crashed?
I looks like from this log that a service is crashing and causing this issue. I do not apply update locks by choice with this container/service but it crashing must be the culprit:
Feb 05 08:18:20 daff6ff resin-supervisor[2315]: [debug] Starting current state report
Feb 05 08:18:21 daff6ff resin-supervisor[2315]: (node:1) UnhandledPromiseRejectionWarning: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com api.balena-clo>
Feb 05 08:18:21 daff6ff resin-supervisor[2315]: at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:56:26)
Feb 05 08:18:21 daff6ff resin-supervisor[2315]: (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by thr>
Feb 05 08:18:21 daff6ff resin-supervisor[2315]: (node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise >
Feb 05 08:18:21 daff6ff resin-supervisor[2315]: [debug] Supervisor API listening on allowed interfaces only
Feb 05 08:18:23 daff6ff resin-supervisor[2315]: [error] LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com api.balena-cloud.>
Feb 05 08:18:23 daff6ff resin-supervisor[2315]: [error] at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:56:26)
Feb 05 08:18:28 daff6ff resin-supervisor[2315]: [event] Event: Device state report failure {"error":{"message":""}}
Feb 05 08:18:30 daff6ff resin-supervisor[2315]: [event] Event: Service exit {"service":{"appId":1555703,"serviceId":389571,"serviceName":"accelerometer",>
Feb 05 08:18:30 daff6ff resin-supervisor[2315]: [error] LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com api.balena-cloud.>
Feb 05 08:18:30 daff6ff resin-supervisor[2315]: [error] at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:56:26)
Feb 05 08:18:30 daff6ff resin-supervisor[2315]: [event] Event: Device state report failure {"error":{"message":""}}
Feb 05 08:18:30 daff6ff resin-supervisor[2315]: [debug] Finished applying target state
Feb 05 08:18:30 daff6ff resin-supervisor[2315]: [success] Device state apply success
Feb 05 08:18:32 daff6ff resin-supervisor[2315]: [event] Event: Device state report failure {"error":{"message":""}}
Feb 05 08:18:34 daff6ff resin-supervisor[2315]: [event] Event: Device state report failure {"error":{"message":""}}
Feb 05 08:18:38 daff6ff resin-supervisor[2315]: (node:1) UnhandledPromiseRejectionWarning: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com api.balena-clo>
Feb 05 08:18:38 daff6ff resin-supervisor[2315]: at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:56:26)
Feb 05 08:18:38 daff6ff resin-supervisor[2315]: (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by thr>
Feb 05 08:18:38 daff6ff resin-supervisor[2315]: [event] Event: Device state report failure {"error":{"message":""}}
Feb 05 08:18:38 daff6ff resin-supervisor[2315]: (node:1) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 2)
Feb 05 08:18:45 daff6ff resin-supervisor[2315]: [event] Event: Service restart {"service":{"appId":1555703,"serviceId":389571,"serviceName":"acceleromete>
Feb 05 08:18:45 daff6ff resin-supervisor[2315]: [error] LogBackend: unexpected error: Error: getaddrinfo EAI_AGAIN api.balena-cloud.com api.balena-cloud.>
Feb 05 08:18:45 daff6ff resin-supervisor[2315]: [error] at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:56:26)
Hello @KingstonSteele,
This looks to be a DNS issue within the local network, where the Supervisor is attempting to determine the IP address of api.balena-cloud.com and is not receiving a response from the DNS server in time. Is DNS currently working on the network where the device in installed?
Best regards,
Heds
Well this is an IoT device that will live in various restaurants and bars so during development I’ve been taking this to various establishments, testing out various wifi. So the wifi is different day to day. Could be a coffee shop, a bar, the office or our shop. It eventually will work and sometimes a reboot seems to fix it.
It seems to be an issue with the accelerometer service. And that issue seems to lie with a library we are using called pigpio. It seems to run the exectuable and the pigpio daemon at the same time, increasing gpu usage and sometimes disconnecting the device. I’m not sure if gpu has anything to do with wifi.
I’m using a raspberry pi 3 but I did find this article which makes me think that when gpu is pushed to the limit, wifi fails.
Hi,
The Wifi and GPU shouldn’t have any crossover on the Pi3. Saying that though, nothing that we’ve seen in this thread seems to suggest that. So far it appears that the problem have been down to the setting of update locks and DNS failure.
I’ll get in contact with the device team to see if they’ve anything to add, but I’m not currently aware of any problems with Wifi and GPIO crossover (or GPU). We’ll let you know as soon as they get back to us, but this will probably be tomorrow as they are based in the EU.
Best regards,
Heds
That would be my first thought as well however that is the common denominator. That is running the pigpio daemon results in this error:
initMboxBlock: init mbox zaps failed
Have you tried altering the memory split to allow more bytes to the GPU? This is done via the host configuration in the dashboard.
You mean this variable?
RESIN_HOST_CONFIG_gpu_mem=396
Yes I did.
Hi @KingstonSteele,
At this point, it’d be really useful if when you see network failures you could get into the device (when it settles down again), and pull the NetworkManager logs as well as output from dmesg?
You can do this by SSHing into the host OS and using journalctl -u NetworkManager.service to get the NetworkManager logs, and dmesg for the kernel logs. If you could supply these to us, it might give us a bit more of an idea of what’s going on. Enabling persistent logs might also be useful if the Wifi fails completely, as on a reboot these logs should still be around. If you go to the balenaCloud dashboard and select the appropriate device, then select ‘Device Configuration’ and enable ‘Enable persistent logging.’, this will store the logs on the state partition at /var/log/journal/<uuid>.
Best regards,
Heds