We have been using balena for a while with our autonomous robot. Recently, we updated from ROS to ROS2. With it, we changed from one monolithic container to twenty smaller containers each containing one node.
Since this update, we sometimes see very large 4G network spikes on bootup. The usage is steadily at a level of around 1600 MB / hr. Obviously this will incur extremely high costs if unresolved. So far, the only way to stop the spikes seems to power off the device and power it on again. This indicates to me some sort of failed data transfer loop.
This morning, one of our devices experienced such a spike during power-up. There was no update to be downloaded for the device, so it was a regular boot. Noticing the spike, I managed to log some packets with tcpdump. It seems that the vast majority of packets is going from our balena device to some private IP:
SOURCE DESTINATION PROTOCOL LENGTH INFO
192.168.1.XXX 10.244.XXX.XXX IPv4 1516 Fragmented IP protocol (proto=UDP 17, off=2960, ID=0f99)
192.168.1.XXX 10.244.XXX.XXX RTPS 484 INFO_TS, DATA, Unknown
192.168.1.XXX 10.244.XXX.XXX RTPS 168 INFO_DST, ACKNACK, Unknown
These packets are sent at around 250 Hz, causing the huge spike in data. Is this IP related to a balena service? It looks like it at least goes outside of the internal network, otherwise it would not incur such high 4G network usage.
We have already contacted our SIM provider – even though nothing changed compared to before the network spikes – and they said they couldn’t find any anomalies.
Going on a slight tangent, we also noticed high network usage when updating devices. Updating only one container, which based on image layer sizes should use around ~30 MB, uses 600 MB of data. Is it possible the same issue is at play here?
Please give us some indication of how to resolve this issue, because it was not present when we were on a single-container deployment, and it seems to me that the (incomplete) data transfer is happening between our device and a balena server. We have also tried using the Private Chat Support button (since we pay for Fleets), but that just triggered an infinite loading icon.
Thanks in advance for your help.
All the best,