Is it possible to upload containers to the Balena registry without triggering a release and then refer to these containers at a later stage?
Use case:
Our CI/CD system produces between 20 different containers for 50 different balena apps and that number will grow. Since we cannot deploy the same container to multiple apps without uploading them for every app during deployment we would like the CI system to pre-load the containers to the Balena repos so that we can refer to them later. This would also allow us to get around the fact that the Balena AWS repo endpoints are in the us-east region which creates significant latency at this scale.
Ideally, we would prefer if Balena could just pull images from our AWS ECR repo when we trigger a release but that does not seem to be possible. The --release-secrets option for balena deploy is just a simpler version of docker pull ; docker push which does not help much.
Second best option would be if we would have a single repo in Balena to which we can upload images so that we can refer to them in later releases.
You can refer to images in the docker-compose file using the image entry. So you CI system can build the images and push them to your registry and reference them in the docker-compose file. An example would be:
This will cause our builder to pull the image eclipse-mosquitto from docker hub, when a release is created and store the image then in our registry as part of the release. If you use balena push instead of balena deploy our cloud builder will perform the pull / push of the docker image. We do this to perform access control from the devices point of view. Do I understand it correctly that you would prefer that our build server does not pull the image from your registry, but the device pulls this image from your registry directly? The only difference in that case that I can see, is that your registry could be in a different region and therefore the latency with your devices could improve when downloading the new image. Could you provide us more information? Is my understanding and my summary correct? Did I miss any other advantages when your devices use your registry directly?
Thank you.
I think this only works for public containers but our containers are in our own private AWS ECR repos. I can provide credentials to the balena cli but then the cli first downloads the containers to my machine and then uploads them back to balena.
What I would like to do is that the CI system builds the images and either pushes the images to Balena or we allow balena to pull our images on deploy.
The only thing I want to upload during a deployment is the docker-compose.yml file.
Not sure if this works to allow balena password-less read-only access if the pull comes from a specific ip address.
I’m using balena deploy to trigger a release and AFAIK the balena-cli will always ensure that the containers are available locally and then upload them to the docker repo for that app.
I would like the balena deploy server to pull our containers from our repo in the cloud and not upload the containers.
We do not currently support pulling directly from private registries. We also only support devices pulling from our registry to enforce access controls. I have opened an internal issue to track that request, and we will of course keep you updated with that feature.
As an alternative, you may be able to balena deploy from your CI system but use pinned releases to ensure that no devices are promoted to the latest release until you are ready. This configuration would prevent locally cloning the images from your ECR to your laptop, and instead leverage your VPC to do the transfers internally.
Thank you for the feedback, and please let us know if you have further questions or we have misunderstood some part of your request!
Thx. If you add support for this also add the ability to share containers between apps. We have one container which needs to be on 50-100 different applications in addition to app specific containers and we have to upload the same container for every app separately.
It would be much more useful to decouple the Balena registry from deployments allowing users to upload containers as they see fit and then check if you have all required containers in your registry when we trigger a deployment (aka upload docker-compose.yml file).
@xginn8 I’d like to add that pinning releases is quite different from decoupling containers and deployments and why we’re not going to use that.
A docker-compose.yml file contains both container references and configuration, e.g. in form of environment variables. Pushing a release to the device might contain the wrong configuration and should never be put on the device. There is no way for us to know that.
Also, since it is not possible to delete releases we cannot purge releases which should never be deployed to the devices.
The current setup is insufficient for this workflow. You need to decouple uploading of artifacts (containers) from the actual deployment (upload of docker-compose.yml) to support this or provide us with a way to give you credentials so that you can pull our containers.
Hey @magiconair,
Thank you for your feedback. We understand how this will be valuable. We have raised an internal issue and will be discussing this with the team. We will let you know once we have more information on this topic.
Context: We have about 100 apps with 2 devices each which we use for factory automation. Each pair runs a specific set of containers and only that pair runs that combination. The number of apps will continue to grow since an app relates to machine in one of our factories. More machines and factories -> more apps.
Here is the gist:
All apps share at least one container (think supervisor). Right now we have to upload the shared container 100 times when we want to update it.
The private registry feature does not solve our problem since I still have to upload the container. I would like Balena to pull the images from our ECR repos so that we don’t have to upload them to Balena every single time. I only want to push a docker-compose.yml
There is no container upload without a deployment. So we cannot pre-emptively upload containers from our CI system and then just refer to them during a deployment.
The app model to define a software package does not fit our use case very well but we could live with that if deployment wouldn’t be so inefficient. It would make more sense if you could either just pull our containers on deploy or we would have a common repo for all of our containers for all apps and then can point to this repo instead of having an independent copy for each app.
Your app model is built for low number of apps with large number of devices or assumes that there is no or little dependency between the apps.
Since an app is the only way to specify which software runs on a device we have to create lots of apps, or build fat binaries, or fat containers which means we install a lot of software we don’t need or want for security reasons.
Hi Frank, I’m sorry I missed some of the earlier messages, thank you for providing this summary!
Regarding your situation, we do have plans on implementing multiapp and optional containers which would certainly help in your case. With multiapp you could theoretically deploy multiple applications to a device, so any shared container would be managed separately. We don’t have however an estimate on when this features will be in production. Much of the foundational work is already in progress or done, but I don’t think it’ll be ready in the short term.
As for the ability to source images from a private ECR, that is something that is scheduled for internal discussion but has not happened yet. I’ll make sure to raise this item again to prioritise the discussion.