I am setting up code that will set up tags that represent the devices internal status. There will be a time of day that the devices come online. One day (hopefully) there will be hundreds of devices that make API calls rapidly at the same time (UTC 6 and 11 and about (10-40 calls each). That day to a lesser extent could be as soon as tomorrow.
Is that going to be a problem and should I rate limit myself a bit, or is this expected behavior? I don’t want to DDOS balena.
I am using the scripts in a slightly modified form from here to do the updates.
hundreds of devices, doing between 10 to 40 calls each at a specific time to the same site pretty sounds like a denial of service attack. (I am not from the balena team).
We do perform some rate-limiting for our API endpoints, but the specific limits are not documented and we are still investigating what the best values would be. But we’ll publish more info about this in our docs a bit later in future.
For your case, we would strongly recommend adding 2 points to the implementation.
Random factor in API call scheduling. For instance, instead of configuring the devices to send an API request exactly at 06:00, you could schedule the calls at a random point of time in a period from 05:50 to 06:10.
Exponential backoff logic. If the devices receive an error from the backend, there should be a delay before next retry, and it should grow with every consequent error response.
Following these best practices will help to ensure your devices behave correctly communicating with balena cloud API.
This is still earlier in my pipeline, there was no retry logic more complicated than try again in 10 minutes. However, I will change that to a random power of 2 with a rising floor.
As for having multiple first attempt point. I think I will just pick a random time within a half hour window for each device.
This isn’t too time sensitive so I will do my best to distribute the load.
