We are experiencing random restarts that with the help of Balena engineers has been traced to a chip that controls USB power. This chip.
This is most likely because we are using 3 of these adapters per a device in a montioring mode which according to the specs draws 290mA during full receive mode.
This should be under the port max of 500mA hours but due to the substained nature of our usecase we are most likely:
Overheating that chip linked above
Overheating the chipset on the adapter causing it to draw more than 500 amps, and flipping current protection on that chip above
The heat is from the chipset on the adapter. (as best I can tell)
Please let me know if I am mistaken in any of my thoughts above, but here are our proposed solutions that has to be in place by next Tuesday.
Hiring a 3d designer to expand the area under the lid (see the image below)
Adding a fan to either a hole I make in the side or a hole in the new lid.
Buying a bunch of these 6-inch USB extension, then plugging the adapters into those and hot glueing them to the side of the case to just reduce the heat passed back from the case to the fin
With the expanded case, putting something like this powered usb hub that would draw from the 5v rail exposed on the Giop pins.
In addition to the fan, add little heat sinks to the back of the USB ports and if we get the right sized thermal pad straight on the chips themselves.
Questions regarding the above.
Would our 3d cases melt? Has anyone done this before? What should we look out for here?
Can I power the fan off of the black pins labeled ext (the ones the left side of the fin if the USB ports are facing down)? I found one with 3.3v and one that was a ground. But I can’t find them in the schematics.
This is a question of if the issue is heat alone or extra amps drawn because of the heat? Other than being really really ugly for production devices is there anything I am missing with this plan?
This might the best one. Can I power this safely of the 5v GIOP pins? Will this put load on something that wasn’t meant to have load put on it? Will heat become an issue in the power management circuits of the fin?
If anyone has anything at all to add this would be great. This power issue only showed itself after putting a lid on the cases so we know we are close to being able to ship.
Thanks for your time and thank you to @chrisys @samothx @ntzovanis
For your many support chat messages that got us this far!
Hello @tacLog thanks for sharing this. Here’re my thoughts on your questions:
There are many plastics used for 3D printing that will serve the purpose. You can take a look here for examples. Look for any that has a glass transition temperature (or “heatproof”) of 70C+
Those headers are exposed from the co-processor (J24 on Sheet 7 of the datasheet). There’s only 3.3V and limited to 100mA. You should use the coloured HAT headers to get 5V with higher current capabilities.
My theory is that it’s just heat that triggers the safety feature. Another test we could try is to put a temperature sensor inside the case and see how hot it gets at full load.
The 5V pins on the HAT header are intended to power external components. There is a maximum current of 2.5A (shared between all the devices connected to 5V) so should be enough for a fan or a USB hub.
We are getting the first batch of the 3d printed cases today. I have no idea what they are printed out of so we should assume the lowest quality plastic out there. I have yet to have them in hand, but my thoughts are to try and melt one to make sure it doesn’t happen in the field. i.e. Put in a fin in the case with the 3d printed lid. Put the whole unit in a box with a jacket over it and run stress to see how hot we can make things.
I figured that out after I used the Co-processor headers and then fan speed was the definition of slow. I had to use some cable extensions but I have them running on the 5v rail now. They are very small and don’t move a lot of air, but they make the difference.
I don’t have a temperature sensor of any kind. I should have ordered one Friday. I generally touch things and if they nearly burn me they are really hot. What I do have is a USB draw reader now. Under my use case, my USB wifi adapters run at .16-.17 amps at 4.77 volts. Please note that because this USB draw reader distances the adapter from the case it reduces the heat load on both the adapter and the chips. I would expect an adapter running hotter to draw more. We notice a 2x difference in data volumes between an adapter running cold or warm, vs hot.
Thanks, my only thought is for the future if we use the fin over a raspberry pie zero w, is that what would get hot if we are drawing 3 or 4 watts from that. My assumption is that it would handle it though.
Thanks for your responses they are very helpful. A general update will follow. We should be shipping with 80 percent confidence tomorrow.
Cut with a Dremel to size. They only are placed on the bottom two, the left one never got past warm to my finger test. I think this is because the heat is mostly from the devices plugged into the USB ports. See photo 1 at the end.
A larger heat sink on the back of the USB ports that takes most of the heat away before it gets to the chips in the above photo.
A fan installed on the left side of the case powered by the 5v rail.
Things I am still worried about:
The coper pads slipping off and shorting out.
The fan failing early and the device becoming completely unstable
The wires getting in the way of the fan after being moved around during shipping
The solution not being enough cooling if I increase CPU load later
The copper pads slipping off and shorting something out.
Reference photo 1: (the copper pads are peaking out from beneath the larger heatsink
Small update. We shipped these our last week. The screw supports were weak and one of them broke form over tightening, but overall they where sufficient for the pinch I was in.
I received the OK to share the design files, and I will post them this week.
They melt if you apply direct heat, so I would not recommend 3d printing the bottom part of the case as the CPU might get hot enough to sag the plastic. However, I couldn’t see how the top would ever get that hot with a fan running.
I wanted to add a small contribution to the overheating of the USB components on the fin 1.1 as well. We use the fin in industrial applications for monitoring and control of systems in relatively hot environments with an ambient around 90 - 115 deg. F.
As the summer has kicked in we noticed failure across all our usb to serial comms and the ethernet adapter as well.
The failure condition we noticed in our /var/logs/syslog was: ERROR::dwc_otg_hcd_urb_enqueue:491: Not connected
Upon reading this post we removed the lid on the fin case and added vents to our waterproof enclosures. We will be adding active cooling once tested as well. This has solved the issue for us.
One other note is that the while all this is happening we still had relatively low processor temps of less than 65 degrees C. We did notice that at about 63 - 65 degrees C processor temp we started to get the USB OTG failure in our logs where serial and ethernet comms would drop until the enclosure was vented.
I’ve just wanted to let you know that we have published our root-cause analysis (RCA) of the thermal issue discussed in this thread. Don’t hesitate to reach out to us if you have any questions about the fix itself or the process to get your units fixed.
