Kubernetes for scaling

mm-e1 · June 28, 2021, 4:23pm

dtischler · June 28, 2021, 4:29pm

teslov · July 12, 2021, 11:04am

Hello, currently I’m developing proxy server with dashboard from Razikus/open-balena-dashboard frontend. It will allow login using web ui, and so on. Also i will add possibility to create health-checks dashboards for all devices in your openBalena instance. Will you adopt my version of proxy for k8s deployment?

bversluijs · July 12, 2021, 6:17pm

Hi,

Nice project you’re working on! I’ve seen it on the forums and it looks promising!
It’s certainly a good addition for openBalena and it is always possible to have it running next to your openBalena instance.

Without looking at the code or project itself, it certainly is possible to integratie it in the Kubernetes Helm Chart. However, because I just wanted to create support for Kubernetes and nothing more than that, I think it’s best not to include it if it isn’t included in openBalena itself at the moment. Just so that the Kubernetes support is just the same as the openBalena deployment.

I’ll definitely provide support if necessary when you want to create a Kubernetes version of your Web UI that integrates well with the openBalena Kubernetes support! And last but not least, keep up the great work!

teslov · July 29, 2021, 7:42pm

Here it is

jose_barrero · May 3, 2022, 3:39pm

hey @bversluijs awesome work!
Im trying to run openbalena k8s on EKS fargate I’m just starting to get familiar with k8s and helm so I’m asking for some guidance here.

I’m simply following the README instructions so what i did was:

clone your fork of the project
run the quickstart script (it finishes with no errors)
create an EKS cluster (does this work on fargate mode?)
eksctl create cluster --name openbalena3-dev --region us-east-2 --fargate
add HAProxy Ingress repository
helm repo add haproxy-ingress https://haproxy-ingress.github.io/charts
install open balena chart

NAME: openbalena
LAST DEPLOYED: Tue May 3 01:14:38 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

After that I see the only service running is the haproxy container, other services have a desired state of 1 but they just don’t start. Also I’m not very clear on how to delete the ingress rule once (i just have to delete the repo using helm?)

Then when i try to install cert manager on my cluster I can’t get to verify the installation, i get the following error:

Not ready: Internal error occurred: failed calling webhook “webhook.cert-manager.io”: failed to call webhook: Post “https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s”: no endpoints available for service “cert-manager-webhook”

I get the same error if I try to install the issuer yaml file from your repo.

bversluijs · May 3, 2022, 5:45pm

Hi Jose,

Thanks for posting your question on the forum.
I don’t have any experience with EKS or EKS with fargate mode, so I can’t help you there.

I have some experience with Kubernetes and it seems like the Helm install was succesful. Can you use the describe command on the pods that aren’t started?

I must say, I have been working on a new version of the Helm chart for quite a while now and I’m waiting for Dependency upgrades by bartversluijs · Pull Request #137 · balena-io/open-balena (github.com) to be merged.

You can, however, already test it out by using the dependency-upgrades branch on my open-balena repository. That Helm chart is much more up to date and much better. It’ll work with Kubernetes 1.22, because the current master Helm chart is quite old. So try that and read the README. Hopefully this will fix most of the problems. Updating this chart is also much easier in the future

jose_barrero · May 5, 2022, 11:43pm

hi, I’m still trying to run on EKS. I could run the pods succesfully when not using fargate mode, now I’m trying to set the acme signed certificates because I’m getting:
DEPTH_ZERO_SELF_SIGNED_CERT: request to https://api.xxxx.com/login_ failed, reason: self signed certificate
every time I try to login
once I install cert manager, create an issuer, instruct ingress resources to use it and request a certificate what else do i need to do to use the certificate with openbalena? do i need to change the values on the certificates section of the config/kubernetes.yml file?

bversluijs · May 6, 2022, 7:43am

Hi Jose,

Please provide some more information, such as the yaml snippet which instructs the Ingress that uses the issuer and the issuer snippet (without secrets of course).
You just have to instruct Ingress the Ingress to use the issuer and the HAProxy Ingress should do the rest!

bversluijs · May 6, 2022, 1:23pm

Update on the Kubernetes chart. The dependency upgrades PR is approved .
This will fix problems with the latest BalenaOS images, like provisioning or development mode.

Because of this, I’ve updated the Helm chart with those changes. If you were using the Helm chart later than 28th of June 2021 (so after this PR), upgrading is easy. Just update the dependencies as stated in the README and upgrade the Helm chart and you’re good to go!
Please always backup your data first, because there’s always a chance that problems may occur during the upgrade!

During the time of the dependency upgrade testing and the approval of the PR, a lot has changed in the open-balena services (API, VPN, registry), but I’ll test out the latest versions asap and create a new PR for that and update the Helm chart as well!

jose_barrero · May 6, 2022, 2:40pm

hey @bversluijs, sure. the snippets are the ones you put on the README file, the only difference is the mail address.

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: openbalena-certificate-issuer
  namespace: default # Use same namespace as installation
spec:
  acme:
    email: yoso1218@dmosoft.com # Will notify you if something goes wrong with the certificates
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: openbalena-certificate-key
    solvers:
    - http01:
        ingress:
          class: openbalena-haproxy # HAProxy Ingress class of openbalena

after creating the issuer resource, at the end of kubernetes.yml I added this:

ingress:
    annotations:
      cert-manager.io/issuer: openbalena-certificate-issuer

and upgraded the chart.

bversluijs · May 6, 2022, 2:53pm

Ah, I remember what’s wrong. In the latest Kubernetes versions, the IngressClass property is introduced, which isn’t enabled by default for HAProxy.

You’ve to add the following to your kubernetes.yaml file:

haproxy:
  controller:
    ingressClassResource:
      enabled: true

This enables the IngressClass resource for HAProxy, which will then handle the cert-manager request to create HTTP01 solvers. Let me know if the cert-manager pods come up

This resource is introduced since Kubernetes v1.18, which is quite some time ago . I’ll enable this by default and merge it to master, so others won’t stumble upon this problem.

Updated the master branch with the changes, which enables it by default

jose_barrero · May 7, 2022, 2:43pm

@bversluijs It worked, awesome work, you saved us so much time. I’ll be testing the openbalena functionalities and let you know if i have issues. I wanted to ask… do you have some kind of general diagram on how the resources are configured? thank you

jose_barrero · June 3, 2022, 8:55pm

hi @bversluijs, does the VPN still use self-signed certificates? i see the api uses the lets encrypt certs but the vpn doesn’t.

bversluijs · June 3, 2022, 9:33pm

Hi Jose,

The VPN still uses self-signed certificates indeed. I don’t have enough knowledge about OpenVPN and the certificates used in order to make it work with Let’s Encrypt or other certificate providers. Also, I don’t know if this is the preferred way for OpenVPN, I can see why self-signed is used, but as said, I don’t have the knowledge

Please let me know if it’s possible (you or anyone) and let me know if you (or someone) has done it already!

About the diagram, I overlooked that question. I don’t know if you’ve already figured it out, if not, please let me know what you want to know!

bversluijs · July 26, 2022, 7:13pm

TL;DR
I’d like to make the Kubernetes Helm chart easier to use and more accessible and flexible, even for openbalena users using Docker and I want you feedback!

Hi all,

As some of you might know, I’m an active user of openbalena. I’ve created a Kubernetes Helm chart for deploying it, which we’re using for our openbalena instance that connects more than a thousand devices. I’m even confident to say that, if you have some experience with Kubernetes, creating an openbalena instance is faster and easier with the Helm chart, but that’s just my opinion. I hope I’m not the only one using it, though

I’m constantly trying to improve the Helm chart and keeping up with all the updates the Balena team creates and following all changes on the API, VPN, database etc, just so I can create a new release for openbalena as well as the Helm chart.

Now, the reason of this post. There are some things on my roadmap that I’d like to improve over time in the Helm chart (any feedback is more than welcome!):

Username & password should be saved in a secret rather than just as an environment variable;
The VPN certificates should be saved in a secret;
The default VPN endpoint should be cloudlink.<domain> (same goes for openbalena itself) (see this blog);
The Helm chart shouldn’t be a fork of the openbalena repository.

That last point, that’s the main reason of my post. I’ve forked the repository because I didn’t have much experience with Kubernetes and openbalena when I started. Now that I know more, I think it’s possible to make a separate repository that only contains the Helm chart of openbalena and is updated there. This makes distribution a lot easier without all the hassle of merging and keeping up to date with the openbalena repository.

Before I do that, I’d like to share how I want to make it easy for anyone to use this Helm chart. You always have to use the quickstart script of openbalena first. But converting this into a Kubernetes-ready deployment shouldn’t be that hard.

Besides, probably many of you already have an openbalena deployment using Docker. Converting that to a Kubernetes deployment is a lot of work and I’d like to change that. This will make scaling your openbalena instance much easier!

My idea was to create a bash script that takes all your environment variables of the activate file and convert them to the values.yaml file, which is needed for the Helm chart. From then on, all changes to your configuration (such as new certificates, number of replicas etc) should be done via the values.yaml file.

I’d like to know anyone’s opinion about this approach. The idea is to separate the Helm chart from the forked openbalena repository and making the Helm chart easier to use and more accessible.

jose_barrero · October 27, 2022, 3:51pm

hi @bversluijs
I’ve been running openbalena on k8s for some time without issues. I’ve been gaining knowledge about helm and k8s so I would be glad to help if you are planning to separate the repo and improve the chart. I think is the right way to do it as it makes the chart updates easier.