Deploy openbalena on rancher v1 environment problems


I am currently trying to run open balena on my rancher environment. I am using the buildin LoadBalancer and a already running certbot container.

My steps:

  • Use quickstart to build a docker-compose.yml (remove cert-provider and ports 80 and 443)
  • Certificates are build with the already running certbot container
  • Use the docker-compose.yml to start the services on the environment
  • I use the buildin LoadBalancer (HAProxy based to redirect http and https api.domain, s3,domain, registry,domain and vpn.domain to the haproxy from openbalena 80 port (443 and 80 are redirected to the haproxy 80)
  • In the openbalena haproxy I used the fullchain.pem and the privkey.pem from the certbot to generate the open-balena.pem (cat fullchain.pem > open-balena.pem && cat privkey.pem >> open-balena.pem)

After that steps I can login with “balena login” but cannot deploy stuff.
The push command stucks on 0% and I get the following error after a while:
Retrying “” after 2.00s (1 of 3) due to: Error: received unexpected HTTP status: 504 Gateway Time-out

On the registry service I get the following errors using journalctl:

  • Jun 15 15:40:09 62f97b3d5ae1 registry[754]: time=“2020-06-15T15:40:09.904833764Z” level=error msg=“response completed with error” err.code=unknown err.detail="s3aws: RequestError: send request failed

  • Jun 15 15:40:09 62f97b3d5ae1 registry[754]: caused by: Put dial tcp i/o timeout" err.message=“unknown error” go.version=go1.11.2 http.request.method=POST http.request.remoteaddr= http.request.uri="/v2/v2/c3f6dc248c3c3b0905516ce23bb7d99f/blobs/uploads/" http.request.useragent=“docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.4.0-101-generic os/linux arch/amd64” http.response.contenttype=“application/json; charset=utf-8” http.response.duration=2m0.912908474s http.response.status=500 http.response.written=104“v2/c3f6dc248c3c3b0905516ce23bb7d99f”

  • Jun 15 15:40:09 62f97b3d5ae1 registry[754]: - - [15/Jun/2020:15:38:08 +0000] “POST /v2/v2/c3f6dc248c3c3b0905516ce23bb7d99f/blobs/uploads/ HTTP/1.1” 500 104 “” “docker/18.06.1-ce go/go1.10.3 git-commit/e68fc7a kernel/4.4.0-101-generic os/linux arch/amd64.4.0-101-generic os/linux arch/amd64”

It seems like the registry cannot upload images to the minio. I am currently not understand why.

Is there something I am doing wrong? It would be awesome if somebody can help or point me to a solution.

Should I use a completely different approach?

Thanks and bye

I’m unfortunately not familiar with how Rancher does things, and this looks like there’s trouble routing “internal” requests between the services. I could curl S3 and Registry just fine (the actual domain was visible in the error log above; I’ve since edited it out of the post). Can you take haproxy out of the picture and replicate the config with Rancher’s primitives?

Thanks for your reply.

We gonna change our whole infrastructure the next time anyway.
Currently I can use a new development server with open balena and ignore the rancher issue for now.

Thank you very much!

Bye Darian