Balena Fin Reset

Yes, I just granted you access to the gateway with UUID: 82005c41c23074476ea0dac2f99f729c

It looks like it reset an hour or so ago. Thank you!

thanks, will look into that device now

(unrelated to the core issue of this thread) I am noticing the device experiences some failures when trying to set new values to the RGB LED. How are you controlling the LED (are you using a library?) and how often do you change values on the RGB LED?

[ 3662.960368] leds pca963x:green: Setting an LED's brightness failed (-5)
[ 3744.300845] i2c i2c-3: sendbytes: NAK bailout.

We write the LED by setting /sys/class/leds/pac963x:{color}/brightness to a value depending on the color and brightness. The LED is written often - typically two to ten times every two minutes and that particular print message also shows up often. The brightness should only be written to a value of 20, so I’m not sure why this message says (-5). Because of this reset issue (and another pressing issue), I haven’t spent any time looking into this yet. The LED is physically changing its color as we expect it to.

your timing on the LED is well within the bus limits (several times a second) so there shouldn’t be any problem with it. -5 is the error code from the driver, I’ll look into what that means while I wait for your device to experience the reboot

When we are able to download data from a bluetooth device, the LED blinks 8 times rapidly every 31ms. That is probably the worst case use of the LED.

that might be beyond the bus capabilities, I suggest increasing the interval if possible. I will also try to determine the exact max frequency you can drive the led at so that you have a clear benchmark to your application

#status Fin customer is experiencing random reboots on their Fins (v1.1.0) that do not seem related to either balenaEngine crashes or v1.1.0 heat issue. Florin could not reproduce locally, so I asked the user if we could inspect one more time a device showing this issue. I did not find any error relatable to the reboot issue, nor I observed any reboot in almost 3 hours of monitoring. WHile monitoring I also noticed some led driver errors and confirmed with the user that they are probably driving some LED effects at a too high frequency that the bus is not capable of supporting. Suggested the user to inrease the interval in order to avoid the error, but this is not related to the reboot issue.

sorry, that was meant to be the current summary of the ticket and not a user-facing comment :slight_smile: I was about to update you with the same info, but you have them already above :slight_smile:

No worries! For completeness sake, I should add - the LED blinks 8x blue when if has successfully downloaded data from the Bluetooth device, and 8x green when the data has been uploaded to our server - both using 31ms (31ms on, 31ms off). The LED physically blinks correctly at that rate, and the gateway generally downloads and uploads the data fine (and blinks the LED correctly) without generating that LED brightness failure message.

I’ll keep an eye on it though to try and figure out if there is a particular application event that correlates to that message. But as you’ve said, I doubt this is related to the reboot.

I saw CPU temperature peaking at 60.7 celsius while I was monitoring the device, and that is unlikely to translate in the 49 celsius ambient temperature that triggers the v1.1.0 usb issue (that would still not explain a reboot) and definetely not even close to the 95 celsius that triggers a CPU cutoff, so I’m prone to exclude both v1.1.0 usb-issue-rca and general high temperature as a possible cause. I have to note that from when I started monitoring to now, the device CPU temperature rose from ~49 celsius to the current steady ~61, so it might need more time monitoring to completely exclude. I’ll keep an eye on the device for a bit more time and see how it develops

Great! I was also wondering about temperature being an issue, but don’t know how to monitor it. I have felt the Pi get pretty hot sometimes - not right now though. Is there a command I can use to read the CPU temperature?

debian rpi/fin base images include by default the videocore libraries you need to get the CPU temperature, so it’s as easy as /opt/vc/bin/vcgencmd measure_temp

I am currently monitoring it via watch -n 2 /opt/vc/bin/vcgencmd measure_temp that executes the command every 2s

more info can be found at the Raspberry Pi official documentation

Thanks!!

FYI - I granted support access to this device for several hours (6 or 9 maybe?) thinking you wanted to make a quick check. If the gateway doesn’t reboot in the next several hours, I will need to extend the support access time.

I see the device settled at ~64 celsius and is not heating up atm. Since no reboot happened, I wasn’t able to grab any additional data about the issue. If you noticed the device running hotter sometimes, it might be worth trying to log the output of /opt/vc/bin/vcgencmd measure_temp every bunch of seconds so that we can get the CPU temperature at the moment of the reboot happening, and either confirm or exclude the reboot is caused by thermal reasons. This way when the next reboot happens, you can ping us again with this additional info - is this something you could do in your application?

Yes, I can add that to our application. I have a local gateway here that I can program immediately. If my local gateway and the gateway you are monitoring don’t reboot today, I can push this new code to another gateway and give you access again tomorrow.

Also, let me know if you can determine a recommended max LED frequency, or if you can figure out what the (-5) means. I really appreciate your help!

FYI I’ll stop monitoring the device in ~15 minutes, so I suggest push the new code as soon as possible so that when/if the reboot happens, you already have this additional info to share which will help us investigating the issue