Hello,
I was doing a lot of test deploying to devices and my limited server space (will extend it in the future) seems to be shrinking pretty fast. I found the culprits to be:
I’m making an assumption that this is the registry service storing all the deployed images. If this is the case, is there some way to safely remove, or “docker prune” them?
Hey there,
New versions of balena supervisor should be removing dangling images. What is the version of your balenaOS, and supervisor?
You can interact with the container engine (BalenaEngine) for issuing a prune, by sshing the hostOS, and running a balena system prune command.
Hi @mikesimos,
From what i understand, you are referring to the actual device and images on it? I was referring to open balena services running on the server, taking up more and more space as i test deploy various apps to devices.
I assume that the space taken up in the screenshot i posted is from the balena/open-balena-registry service saving/storing all the images i deploy. If this is the case, i would like to know if it is possible to remove them safely without crashing the open balena services.
Hey there.
In that case you can just issue a docker system prune command, on the OpenBalena host, as /var/lib/docker contains the OpenBalena component images (so removing the dangling images after an update should release some space).
At the time, the registry service images are being stored on the bundled S3 service data volume. Removing previous release images is currently not supported, (as it would currently break the capability of pinning a device to a past release). However, you can interact and issue delete requests to the OpenBalena registry for removing images, and reclaiming space More info. Hope that helps!
Thanks for the suggestion.
Two questions:
If i delete the application through balena cli balena app rm someapp, this should delete all the image releases of that app in the registry? I tried this and it did not free up any space.
I tried calling the balena registry api from inside the registry container by curl -X GET http://localhost/v2/_catalog , but i got:
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"registry","Class":"","Name":"catalog","Action":"*"}]}]}
From what i understand i need some kind of a token, but i’m not quite sure how i’m supposed to get it. Any tips?
Edit: i tried generating a key with balena cli: balena api-key generate something and used the generated key in the api call: curl -i -H "Authorization: Bearer 3ECEqKap3zj9zrWIUFkiRvCEZznUNvPA" http://localhost/v2/_catalog, but still got an UNAUTHORIZED response.
Hello,
Deleting an application deletes the releases on the API but not the image data on the registry.
Registry’s _catalog endpoint is not accessible. You may find what images are stored on the registry by querying the API’s image resource.
To get a token for the registry see this document: https://docs.docker.com/registry/spec/auth/token/
Thank you, will try this out.
Hm, so i tried a few things, but i think i’m a bit confused as to what " by querying the API’s image resource." means. Docker documentation does not seem to have any api paths that have image in them. So i assume you meant:
/v2/<name>/tags/list
balena images
registry.myserver.net/v2/3b7ab6ae3a1edf9c639df864dfcae862
curl -i http://localhost/v2/registry.myserver.net/v2/3b7ab6ae3a1edf9c639df864dfcae862/tags/list
Www-Authenticate: Bearer realm="https://api.myserver.net/auth/v1/token",service="registry.myserver.net",scope="repository:registry.myserver.net/v2/3b7ab6ae3a1edf9c639df864dfcae862:pull"
balena api-key generate something
curl -k "https://api.myserver.net/auth/v1/token?service=registry.myserver.net&scope=repository:registry.myserver.net/v2/3b7ab6ae3a1edf9c639df864dfcae862:pull" -H "Content-Type: application/json" -H "Authorization: Bearer aPMAgbUgkI5pp786gFuNnCJolSwaDvoR"
curl -i -H "Authorization: Bearer eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IlhKTEg6TzRUTjpRUFZQOkNISVM6UVpWVzpXSVI3OlFBRVg6N01XQzpKVFBDOkhPMko6N01SDzpTTQJRIn0.eyJqdGkiOiI4MmEzYzRmOS03ZDg0LTQ4ZWQtYWYzNC01MzE4ODBmNGI3ZmYiLCJuYmYiOjE1NzQ2NjU5NTAsImFjY2VzcyI6W10sImlhdCI6MTU3NDY2NTk2MCwiZXhwIjoxNTc0NjgwMzYwLCJhdWQiOiJyZWdpc3RyeS5pYW11cy5ldSIsImlzcyI6ImFwaS5pYW11cy5ldSIsInN1YiI6InJvb3QifQ.NvzEeRDC2Ww_1838tNiaCeya5tKork-1bOf8y4sYVuuyFe4-yFl7Zufzv7ibdmObyLj53w77KeE9WYosptMbOA" http://localhost/v2/registry.myserver.net/v2/3b7ab6ae3a1edf9c639df864dfcae862/tags/list
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"registry.myserver.net/v2/3b7ab6ae3a1edf9c639df864dfcae862","Action":"pull"}]}]}
curl -i -H "Authorization: Bearer eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IlhKTEg6TzRUTjpRUFZQOkNISVM6UVpWVzpXSVI3OlFBRVg6N01XQzpKVFBDOkhPMko6N01SDzpTTQJRIn0.eyJqdGkiOiI4MmEzYzRmOS03ZDg0LTQ4ZWQtYWYzNC01MzE4ODBmNGI3ZmYiLCJuYmYiOjE1NzQ2NjU5NTAsImFjY2VzcyI6W10sImlhdCI6MTU3NDY2NTk2MCwiZXhwIjoxNTc0NjgwMzYwLCJhdWQiOiJyZWdpc3RyeS5pYW11cy5ldSIsImlzcyI6ImFwaS5pYW11cy5ldSIsInN1YiI6InJvb3QifQ.NvzEeRDC2Ww_1838tNiaCeya5tKork-1bOf8y4sYVuuyFe4-yFl7Zufzv7ibdmObyLj53w77KeE9WYosptMbOA" http://localhost/v2/
Which returns an 200 OK response.
Any suggestions as to what i might be doing wrong? Do i maybe need to escape the slashes and periods in the repository name when generating tokens and querying the docker registry?
Hi,
You can try to query the image resource on the balena API. So curl -H "Authorization: Bearer $TOKEN" https://api.myserver.net/v5/image should give you the image resources, that are available in the API. Each entry there should have a property called is_stored_at__image_location which holds the name of the docker image in the registry.
Hi, thanks for the answer
When i query this api, the is_stored_at__image_location returns names like this:
registry.myserver.net/v2/0143e850de5a54e920e299f8673eb257
A bit bellow this it has a property called content_hash, which has something like this:
sha256:e8d13a27a28be7bd41f37c1f6dd65ed028c05322243a1b0ba76cfc4a0699b797
Is this the “digest” that docker api is refering to as <reference> here?:
DELETE /v2/<name>/manifests/<reference> Manifest Delete the manifest identified by name and reference. Note that a manifest can only be deleted by digest
If so, would i call it like this?:
/v2/registry.myserver.net/v2/0143e850de5a54e920e299f8673eb257/manifests/sha256:e8d13a27a28be7bd41f37c1f6dd65ed028c05322243a1b0ba76cfc4a0699b797
And full:
curl -X DELETE -i -H "Authorization: Bearer eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IlhKTEg6TzRUTjpRUFZQOkNISVM6UVpWVzpXSVI3OlFBRVg6N01XQzpKVFBDOkhPMko6N01SDzpTTQJRIn0.eyJqdGkiOiI4MmEzYzRmOS03ZDg0LTQ4ZWQtYWYzNC01MzE4ODBmNGI3ZmYiLCJuYmYiOjE1NzQ2NjU5NTAsImFjY2VzcyI6W10sImlhdCI6MTU3NDY2NTk2MCwiZXhwIjoxNTc0NjgwMzYwLCJhdWQiOiJyZWdpc3RyeS5pYW11cy5ldSIsImlzcyI6ImFwaS5pYW11cy5ldSIsInN1YiI6InJvb3QifQ.NvzEeRDC2Ww_1838tNiaCeya5tKork-1bOf8y4sYVuuyFe4-yFl7Zufzv7ibdmObyLj53w77KeE9WYosptMbOA" http://localhost/v2/registry.myserver.net/v2/0143e850de5a54e920e299f8673eb257/manifests/sha256:e8d13a27a28be7bd41f37c1f6dd65ed028c05322243a1b0ba76cfc4a0699b797
I tried this out, by generating a token for delete action, but again got an UNAUTHORIZED error. A problem i noticed is that i can’t seem to generate any token that let’s me do any action on the balena registry docker api, i can only check the default /v2/ endpoint without problem with the generated token.
Hi,
I’m not sure how you generated the JWT in this case, but in general you should be able to do a docker login registry.myserver.net for the registry and use your username and as password the api key you generated from balena api-key generate something.
I’m not 100% sure if the content_hash is the reference in the registry API, but to get the reference you can query the registry like this: GET /v2/<name>/manifests/<tag>, the reference is the value of the Docker-Content-Digest header in the response. To get the list of available tags for a given name, you can use GET /v2/<name>/tags/list on the registry.
The name of the image in the registry in this example is v2/0143e850de5a54e920e299f8673eb257. The registry.myserver.net/ in front means in docker term, that it uses a private registry hosted @ registry.myserver.net and not docker hub.
Sorry for another wall of text 
docker login https://registry.myserver.net
I got two different responses on two different machines:
return: Error response from daemon: Get https://registry.myserver.net/v2/: x509: certificate signed by unknown authority
return: Error response from daemon: Get https://registry.myserver.net/v2/: error parsing HTTP 400 response body: invalid character 'I' looking for beginning of value: "Invalid scope"
I found that one can add --insecure-registry, but docker says that there is no such flag.
curl -u something:43DtE5kH0qmpOWDwlS5phg218cyMn23y -vk -X DELETE https://registry.myserver.net/v2/0143e850de5a54e920e299f8673eb257/manifests/sha256:e8d13a27a28be7Gd41f37c1f6dd65ed028c05322243a1b0ba76cfc4a0699b797
Since the api is https://registry.myserver.net/v2/ i was not sure if i need to also put the v2 from the image name, but i tried that also:
curl -u something:43DtE5kH0qmpOWDwlS5phg218cyMn23y -vk -X DELETE https://registry.myserver.net/v2/v2/0143e850de5a54e920e299f8673eb257/manifests/sha256:e8d13a27a28be7Gd41f37c1f6dd65ed028c05322243a1b0ba76cfc4a0699b797
return: { "errors": [ { "code": "UNAUTHORIZED", "message": "authentication required", "detail": [ { "Type": "repository", "Class": "", "Name": "0143e850de5a54e920e299f8673eb257", "Action": "delete" } ] } ] }
header return: Bearer realm="https://api.myserver.net/auth/v1/token",service="registry.myserver.net",scope="repository:0143e850de5a54e920e299f8673eb257:delete",error="insufficient_scope"
balena api-key generate something
return: ixnrIEcyUyuU3To6IShprrJgvTn41nRV
curl -k "https://api.myserver.net/auth/v1/token?service=registry.myserver.net&scope=repository:0143e850de5a54e920e299f8673eb257:pull" -H "Content-Type: application/json" -H "Authorization: Bearer ixnrIEcyUyuU3To6IShprrJgvTn41nRV"
return:eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IlhKTEg6TzRUTjpRUFZQOkNISVM6UVpWVzpXSVI3OlFBSVg6N01XQzpKVFBDOkhPMko6N01EMzpTTUJRIn0.eyJqdGkiOiJlMDRmZGY3Yy04ZmVmLTRmN2ItYTdhYy01MjlmNjgwNzU3NmIiLCJuYmAiOjE1NzQ2ODc0MjksImFjY2VzcyI6W10sImlhdCI6MTU3NDY4NzQzOSwiZXhwIjoxNTc0NzAxODM5LCJhdWQiOiJyZWdpc3RyeS5pYW11cy5ldSIsImlzcyI6ImFwaS5pYW11cy5ldSIsInN1YiI6InJvb3QifQ.09nS9ZwLaL_bkCwv6N00uIh4SQ2Rmp6GiWpRPQ-qKilk5nQmeR5BR_H3XZEwrVkOsn0CDc886kfUynP5eXkBiw
With this token i:
curl -X DELETE -i -H "Authorization: Bearer eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6IlhKTEg6TzRUTjpRUFZQOkNISVM6UVpWVzpXSVI3OlFBRVg6N01XQzpKVFBDOkhPMko6N01SDzpTTQJRIn0.eyJqdGkiOiI4MmEzYzRmOS03ZDg0LTQ4ZWQtYWYzNC01MzE4ODBmNGI3ZmYiLCJuYmYiOjE1NzQ2NjU5NTAsImFjY2VzcyI6W10sImlhdCI6MTU3NDY2NTk2MCwiZXhwIjoxNTc0NjgwMzYwLCJhdWQiOiJyZWdpc3RyeS5pYW11cy5ldSIsImlzcyI6ImFwaS5pYW11cy5ldSIsInN1YiI6InJvb3QifQ.NvzEeRDC2Ww_1838tNiaCeya5tKork-1bOf8y4sYVuuyFe4-yFl7Zufzv7ibdmObyLj53w77KeE9WYosptMbOA" http://registry.myserver.net/v2/0143e850de5a54e920e299f8673eb257/manifests/sha256:e8d13a27a28be7bd41f37c1f6dd65ed028c05322243a1b0ba76cfc4a0699b797
return: { "errors": [ { "code": "UNAUTHORIZED", "message": "authentication required", "detail": [ { "Type": "repository", "Class": "", "Name": "0143e850de5a54e920e299f8673eb257", "Action": "delete" } ] } ] }
header return: Bearer realm="https://api.myserver.net/auth/v1/token",service="registry.myserver.net",scope="repository:0143e850de5a54e920e299f8673eb257:delete",error="insufficient_scope"
Is it possible the balena cli user does not have the needed permissions to communicate with the registry?
I found in one forum:
Only the system administrators have the privilege to catalog repositories, so you need to request the token with system admin users
Is the user generated when following the https://www.balena.io/open/docs/getting-started/ tutorial an administrator?
Hi,
Sorry about the docker login stuff, I wanted to remove this before posting the response, but I forgot.
I looked at it in more depth, and unfortunately I realized that deleting images from the registry won’t work easily out of the box. The reason for this is the following: To delete images from the registry the registry requires the delete action. We use our API as authentication and authorization server for the registry and our API does not handle the delete action at all at the moment. I thought it might be possible to add another authentication source to the registry, but I also looked into the registry configuration for the authentication section (https://docs.docker.com/registry/configuration/#auth), and unfortunately one can only register a single authentication provider.
Thanks for taking the time and effort in helping me! I guess i will just leave it be for the moment and will try to not waste any deployments for uncritical things… Maybe also setup a domain for a local open balena instance for test deploying 
.
I have a same question, I have openBalena running on a server, and I’d like to clean the registry. I don’t care about pinning etc, I can just deploy a new one.
If there is no way to cherry pick what to delete. Can I instead nuke the whole registry easily (and still have functionin openBalena)?
Hi @Ciantic, there is quite a lot of information in this thread, and my understanding from the last message from afitzek is that this is not currently possible with openBalena Pruning balena registry stored images. Have you tried anything and are having similar issues to those detailed above?
@garethtdavies I don’t have login problems. My question is the original in the thread: How do I clean the docker registery created by openBalena’s docker-compose. I don’t need to cherry pick images, I could just clean the whole registry.
Can I just delete the volume: “openbalena_registry” (seen with docker volume ls).
Hi @Ciantic. If I understand you correctly that you just want to clean the whole registry then according to @mikesimos’ advice you should be able to issue docker system prune on the OpenBalena host. Is there a reason that this wouldn’t work for you?
Thanks,
James.
I know this is an old thread, but just want to inform the community that the latest release of open-balena-admin now has the ability to delete releases, which handles a bunch of cleanup, the most important of which happens in the registry:
This was critical functionality for us as without this, the openbalena registry perpetually expands in size. The releases page of open-balena-admin has also been modified to show which devices are running a release, and also any tags for that release, to help decide which releases are safe to delete (we’ll eventually want to build in validation that prevents you from deleting releases which have active devices - so for now just be careful!).
Once you delete the desired releases (which takes a bit of time, in my experience approx 30 seconds per release) you’ll then want to ssh on to your registry container and run the following command:
/usr/local/bin/docker-registry garbage-collect /etc/docker-registry.yml --delete-untagged
This will delete any blobs that are no longer needed after the releases have been deleted, freeing up the space and allowing for better long term maintenance of openbalena. Hopefully others find this helpful too, and as always I’d be interested in any feedback on how to improve it.
Is anything required to access this expanded capability? After updating to a recent version (May 11) and confirming that all prior functionality works, the Release list looks the same as before.