Getting started - Cannot Push Images to registry

openBalena
1

Hi there!

I’m a bit stuck in the getting started guide at the following point:

FROM balenalib/%%BALENA_MACHINE_NAME%%-alpine
CMD [ "balena-idle" ]

balena deploy myApp --logs

I cannot get balena to push the image to the registry, it keeps trying and then fails after 3 tries.

I then took a look at the logs of the registry, and i always see the following lines:

Jul 26 15:07:05 43e54771f23a registry[1211]: time=“2022-07-26T15:07:05.878061021Z” level=error msg=“response completed with error” auth.user.name=admin err.code=unknown err.detail="s3aws: InvalidAccessKeyId: The Access Key Id you provided does not exist in our records.

As well as

Jul 26 15:07:05 43e54771f23a registry[1211]: status code: 403, request id: 1705697D96AE99DB, host id: " err.message=“unknown error” go.version=go1.16.15 http.request.host=registry.domain.com http.request.id=15171107-86a9-48ff-a96d-ed278f6a47c4 http.request.method=POST http.request.remoteaddr=xxx.xx.xx.xxx

I also tried it from the host itself with balena cli and the same result occured.

I am not sure if the Log Entry is related to my problem, but it only appears while trying to push the image to the registry.

Balena-CLI version: 13.10.0
Docker Versions of Open-Balena:

balena/open-balena-s3:v2.13.8
balena/open-balena-db:v5.1.0
balena/open-balena-registry:v2.25.3
balena/open-balena-api:v0.209.2
balena/open-balena-vpn:v11.4.6

I do not intend to use s3aws Storage, but i did not come accross a Point in the setup to define to use local storage.

Can anyone point me in the right direction?

2

Hi Lukas, what kind of device are you using?

4

Hi Julia

The Server runs on a virtual Ubuntu 20.04, in my fleet I have raspberry pi 3b.

As local machine i first tried with Ubuntu as WSL on Windows 11, afterwards I installed the balena-cli on the same System as the balena-os dockers

5

Any solution to this? My understanding is that by default, openBalena uses Minio storage and not s3aws. I don’t seem to see where to change to use the local s3 container as a server. Please help!

6

Hi,

Openbalena uses Minio as a standard S3 provider. S3 is from AWS and that’s why the error indicates it uses AWS, but instead it uses the Minio instance provided by open-balena-s3. But maybe you knew that already!

It looks like the registry can’t connect with the right credentials to the Minio server. I can’t debug your instance, but what I can do is spin up my own instance and try if I can replicate the issue. So give me some time and I’ll report back here!

7

Hey :slight_smile:
That would be great, I already spun up a second Openbalena on a different Server and Hypervisor with the same results. But it would certainly help if someone with more knowledge on the subject would try it as well and maybe find a solution

8

So I’ve tested it and replicated the problem.
The problem seems to lie in the fact that Minio has changed something in the latest releases, which breaks the open-balena-s3 container. However, it seems like that is fixed in v2.13.9, which pins the Minio release.

So my suggestion would be to upgrade the container to the latest version. I’ll also create a PR that does this for the open-balena repository, so you can wait for that or change the version in the compose/versions file manually for the time being.

Please let me know if this helps!
Maybe you should remove the S3 volume, but try to upgrade first! :slight_smile:

Created open-balena PR #158

The PR is merged by the way! (PR #159)
Updating to openbalena v3.7.5 will most likely fix the issue.

9

Hey :slight_smile:
Thanks for debugging and resolving the issue. I had to remove the S3 volume, only upgrading didn’t quite do the trick. After removing the S3 volume and restarting compose everything worked as expected.

Greetings
Lukas