I randomly happened upon this article on the Google Cloud Platform blog:
It refers to an announcement by Docker stating that anonymous docker hub pulls will shortly be limited:
Earlier this summer, Docker announced it will begin rate-limiting the number of pull requests to the service by “Free Plan” users. For pull requests by anonymous users this limit is now 100 pull requests per 6 hours; authenticated users have a limit of 200 pull requests per 6 hours. When the new rate limits take effect on November 1st, they might disrupt your automated build and deployment processes on Cloud Build or how you deploy artifacts to Google Kubernetes Engine (GKE), Cloud Run or App Engine Flex from Docker Hub.
At which point in the balena deploy process are images pulled from Docker Hub?
Purely speculation, but it feels to me like those limits are never going to be hit, except by the most active development efforts and maybe the most rogue running applications. Even when I’m doing rapid-fire updates, trying to figure something out, I can’t do it for more than a few hours at a time, and I don’t come anywhere near 200 pulls in that time. I do get up to maybe 50-100 pulls.
Professionally-run applications should be leveraging an internal Docker registry for all containers, and not relying on Docker Hub for anything except the containers needed at Docker image build time. That being said, I know that every time I ‘balena push’, that build is being run on balena-managed servers. While a single build is only goinfg to pul a few images, those servers are running builds for a large number of people, likely all on the same authentication credentials (or anonymous) to Docker Hub.
Probably what balena will have to do is tie individual credentials to each build, so that the limits apply per user, rather than the entire balena build infrastructure. It seems like that would be pretty easy to do, by automating an account setup at Docker Hub per balenaCloud login, or asking users for their Docker Hub (API?) credentials.
Good question, @crbn60, this is something we have been discussing internally after these changes were announced back in August.
You’re spot on here, to date these our builders have been anonymous and so requests to Docker Hub from the builder would have been effected by the introduction of rate limiting. We’ll be switching our builders not only to make authenticated pulls but to do so using a paid plan, so they’ll ultimately be unaffected by this change. With this, the builders will continue to function in the same way they have always done.
I just got this error after initiating a balenaCloud build.
[Error] Error: toomanyrequests: You have reached your pull rate limit.
You may increase the limit by authenticating and upgrading:
https://www.docker.com/increase-rate-limit
Note from the referenced page:
Current Docker Hub Usage Limit Status (updated 11/2,4:00pm Pacific)
Unauthenticated requests: 5,000 per six hours
Free tier requests: 5,000 per six hours
Temporary full enforcement window
(100 per six hours for unauthenticated requests,
200 per six hours for free accounts):
November 4, 9am-12 noon Pacific Time.
Looks like we are in the “Full Enforcement” window right now.
Thanks for the workaround suggestion! I was thinking about trying the local build route. Might be good time to learn! It is so easy to build on balenaCloud… I’ve never felt the need to try building locally.
//Sam
