API Rate limits and workarounds

Currently we have around two dozen devices and we are polling each of them via the sdk/api about every 5 seconds.

We’re experiencing a lot of timeout requests and error messages saying there are too many requests. We can’t build reliably anymore.

I was wondering if there is any detailed information on the performance of the api, how it rate limits and when. Would it be a matter of signing up for different plan to get better performance?

Thanks a lot for the help!

Hi there,
We do perform some rate-limiting for our API endpoints, but the specific limits are not documented, and we are still investigating what the best values would be.
May I ask you why you need to poll each one of them every 5 seconds? I’m asking because If we can better understand the use case maybe we can help you find another solution. May I also ask you if in your case making a single request for multiple devices would still be fine ?

Hi @JSReds - Thanks for the suggestion. I’m currently working on an implementation that calls the getAll* versions of the functions instead of multiple get functions.

We are observing the same problem. Our automated tests automatically deploy releases and test configurations by updating and checking environment variables for multiple devices. This usually works well if executed from a developer’s computer, but we are getting the “too many requests” error if executing the tests from the gitlab CI. If not documented, can you provide some indication how the get more reliable results?

Hey Andreas,

it would be faster to fetch environment variables of all of the devices in one query to reduce the number of api calls. I am assuming you are getting too many requests issue on every test run, can you be certain they are only run once? can you approximate how many api calls you might be making? the rate limiting is quite high so you shouldn’t be hitting them for normal use cases.

We are using “balena deploy” via balena CLI to update two applications with about 10 services each. I assume the command line tool will produce API calls. Next, we are using the python SDK and the get_all versions to to obtain the environment variables, for example balena.models.environment_variables.device_service_environment_variable.get_all I expect this is one API call. The test breaks when we create/update environment variables. We do one call per variable, in total about 50 over two applications and 6 devices. We are seeing the errors after about 40 calls. My understanding is there is no api call to set multiple variables?

The problem disappeared for us.

Hi

This is strange. Can you let us know if you start seeing this again?

Would help if you can share some logs - and if possible a script to reproduce this issue. Would make it easier to recreate on our end

Yes, I will let you know if we see it again. I found that we actually do a lot more the 40 API calls. We are doing a balena deploy and then query the devices every second if they are on the right commit and the supervisor is idle. In API calls this is

2021-03-16 11:56:27,651 [DEBUG] caddeploy.device.CADAMI-Server: [WAIT] - Not on right commit or supervisor not idle.
2021-03-16 11:56:28,657 [DEBUG] urllib3.connectionpool: Starting new HTTPS connection (1): api.balena-cloud.com:443
2021-03-16 11:56:28,738 [DEBUG] urllib3.connectionpool: [https://api.balena-cloud.com:443](https://api.balena-cloud.com/) "GET /v5/device?$filter=uuid%20eq%20'dDEVICE_ID' HTTP/1.1" 200 None
2021-03-16 11:56:28,744 [DEBUG] urllib3.connectionpool: Starting new HTTPS connection (1): api.balena-cloud.com:443
2021-03-16 11:56:28,827 [DEBUG] urllib3.connectionpool: [https://api.balena-cloud.com:443](https://api.balena-cloud.com/) "GET /v5/device?$filter=uuid%20eq%20'DEVICE_ID' HTTP/1.1" 200 None
2021-03-16 11:56:28,834 [DEBUG] urllib3.connectionpool: Starting new HTTPS connection (1): api.balena-cloud.com:443
2021-03-16 11:56:28,909 [DEBUG] urllib3.connectionpool: [https://api.balena-cloud.com:443](https://api.balena-cloud.com/) "GET /v5/application?$filter=id%20eq%20'APP_IDb' HTTP/1.1" 200 633
2021-03-16 11:56:28,916 [DEBUG] urllib3.connectionpool: Starting new HTTPS connection (1): api.balena-cloud.com:443
2021-03-16 11:56:28,999 [DEBUG] urllib3.connectionpool: [https://api.balena-cloud.com:443](https://api.balena-cloud.com/) "GET /v5/application?$filter=app_name%20eq%20'OTAcast-server-test' HTTP/1.1" 200 633
2021-03-16 11:56:29,008 [DEBUG] urllib3.connectionpool: Starting new HTTPS connection (1): api.balena-cloud.com:443
2021-03-16 11:56:29,448 [DEBUG] urllib3.connectionpool: [https://api.balena-cloud.com:443](https://api.balena-cloud.com/) "POST /supervisor/v1/device HTTP/1.1" 200 370

Traceback if the error happens

Traceback (most recent call last):
  File "/cadedge_tools/caddeploy_pkg/caddeploy/utils.py", line 66, in run_parallel
    return_values.append(task.result())
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/cadedge_tools/caddeploy_pkg/caddeploy/utils.py", line 23, in wrapper
    return func(obj, *args, **kwargs)
  File "/cadedge_tools/caddeploy_pkg/caddeploy/device.py", line 150, in wait_for_release
    while retries > 0 and not (self.balena.is_on_right_commit() and self.balena.is_idle()):
  File "/cadedge_tools/caddeploy_pkg/caddeploy/balena_device_adapter.py", line 153, in is_on_right_commit
    app_commit = self.balena.models.application.get(self.application_name)["commit"]
  File "/usr/local/lib/python3.9/site-packages/balena/models/application.py", line 146, in get
    apps = self.base_request.request(
  File "/usr/local/lib/python3.9/site-packages/balena/base_request.py", line 197, in request
    raise exceptions.RequestError(response._content)
balena.exceptions.RequestError: b'Too Many Requests'