Getting started - Cannot Push Images to registry

Hi there!

I’m a bit stuck in the getting started guide at the following point:

FROM balenalib/%%BALENA_MACHINE_NAME%%-alpine
CMD [ "balena-idle" ]

balena deploy myApp --logs

I cannot get balena to push the image to the registry, it keeps trying and then fails after 3 tries.

I then took a look at the logs of the registry, and i always see the following lines:

Jul 26 15:07:05 43e54771f23a registry[1211]: time=“2022-07-26T15:07:05.878061021Z” level=error msg=“response completed with error” auth.user.name=admin err.code=unknown err.detail="s3aws: InvalidAccessKeyId: The Access Key Id you provided does not exist in our records.

As well as

Jul 26 15:07:05 43e54771f23a registry[1211]: status code: 403, request id: 1705697D96AE99DB, host id: " err.message=“unknown error” go.version=go1.16.15 http.request.host=registry.domain.com http.request.id=15171107-86a9-48ff-a96d-ed278f6a47c4 http.request.method=POST http.request.remoteaddr=xxx.xx.xx.xxx

I also tried it from the host itself with balena cli and the same result occured.

I am not sure if the Log Entry is related to my problem, but it only appears while trying to push the image to the registry.

Balena-CLI version: 13.10.0
Docker Versions of Open-Balena:

balena/open-balena-s3:v2.13.8
balena/open-balena-db:v5.1.0
balena/open-balena-registry:v2.25.3
balena/open-balena-api:v0.209.2
balena/open-balena-vpn:v11.4.6

I do not intend to use s3aws Storage, but i did not come accross a Point in the setup to define to use local storage.

Can anyone point me in the right direction?

Hi Lukas, what kind of device are you using?

Hi Julia

The Server runs on a virtual Ubuntu 20.04, in my fleet I have raspberry pi 3b.

As local machine i first tried with Ubuntu as WSL on Windows 11, afterwards I installed the balena-cli on the same System as the balena-os dockers

Any solution to this? My understanding is that by default, openBalena uses Minio storage and not s3aws. I don’t seem to see where to change to use the local s3 container as a server. Please help!

Hi,

Openbalena uses Minio as a standard S3 provider. S3 is from AWS and that’s why the error indicates it uses AWS, but instead it uses the Minio instance provided by open-balena-s3. But maybe you knew that already!

It looks like the registry can’t connect with the right credentials to the Minio server. I can’t debug your instance, but what I can do is spin up my own instance and try if I can replicate the issue. So give me some time and I’ll report back here!

Hey :slight_smile:
That would be great, I already spun up a second Openbalena on a different Server and Hypervisor with the same results. But it would certainly help if someone with more knowledge on the subject would try it as well and maybe find a solution

So I’ve tested it and replicated the problem.
The problem seems to lie in the fact that Minio has changed something in the latest releases, which breaks the open-balena-s3 container. However, it seems like that is fixed in v2.13.9, which pins the Minio release.

So my suggestion would be to upgrade the container to the latest version. I’ll also create a PR that does this for the open-balena repository, so you can wait for that or change the version in the compose/versions file manually for the time being.

Please let me know if this helps!
Maybe you should remove the S3 volume, but try to upgrade first! :slight_smile:


Created open-balena PR #158


The PR is merged by the way! (PR #159)
Updating to openbalena v3.7.5 will most likely fix the issue.

Hey :slight_smile:
Thanks for debugging and resolving the issue. I had to remove the S3 volume, only upgrading didn’t quite do the trick. After removing the S3 volume and restarting compose everything worked as expected.

Greetings
Lukas