I think I’ve done it. Finally.
Disclaimer
It works on DigitalOcean, because they have a fix for issue #66607.
It probably works on AWS too, because they also have this fix. But haven’t tested it out.
I’ve created an Helm chart for open-balena, which creates all deployments, services and volumes. It also creates an NGINX Ingress for routing, as well as a HAProxy Ingress for the VPN routing. This means, it created 2 LoadBalancers. It’s necessary. Believe me.
I tried to create it that it’s easy to use if you know how to set-up the Docker open-balena.
You first have to execute the quickstart like you’d normally do. Use an existing email for the cert-manager.
$ ./scripts/quickstart -U <user-email> -P <password> -d <domain>
Then, it creates the settings.yaml in config/k8s/settings.yaml as well as the normal Docker config (config/activate).
Then you’ve to start installing it on your cluster like so:
./scripts/k8s install
This will install the open-balena chart, as well as cert-manager. Which handles your Let’s Encrypt certificates, via HTTP-01 in combination with the Ingress controller. Your super-user username will be used for emailing you if your certificates are about to expire but it can’t renew them.
After this command is done, everything is up and running on your cluster. You’re not done yet!
You haven’t set the DNS settings yet. But because you have 2 LoadBalancers, the DNS settings are somewhat changed. If you’re using DigitalOcean, they LoadBalancers are named k8s-openbalena and k8s-openbalena-vpn (guess which one is for the VPN).
So link the k8s-openbalena to the following domain names:
<yourdomain>
api.<yourdomain>
s3.<yourdomain>
registry.<yourdomain>
Link the k8s-openbalena-vpn to the following domain:
vpn.<yourdomain>
You’re all set. After a few minutes, the cert-manager will have the certificates ready and applied to the NGINX Ingress controller.
Upgrading / applying other settings can be done like so:
./scripts/k8s upgrade
Last, but not least, take a look at config/k8s/settings.yaml.
All your settings are placed here. You can change the replicas per deployment, the storage size per PersistentVolumeClaim, set Sentry DNS’s, use an external S3 storage and check out your username / password if you forgot it.
I’ll be using this in the next couple of days, and notice that the k8s support is not official. Most likely there’ll be changes about the k8s support and settings. So keep that in mind.
Known issues
- The API pod doesn’t run on the first boot. This is because it’s booted faster than the database pod and error’s with an CONNECTION_REFUSED. The pod doesn’t crash however and it doesn’t try to reconnect again. I don’t know why the pod doesn’t crash or reconnect, so I’m creating an issue for that in the open-balena-api repo #383 - Crash on fatal errors. So after everything has spun up, delete the API pod, because this will create a new one which connects to the database.