# Kubernetes for scaling **URL:** \ **Category:** openBalena\ **Created:** [June 12, 2020, 8:35pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102 "2020-06-12T20:35:18Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 12, 2020, 8:35pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/1 "2020-06-12T20:35:18Z")
Hi all, I’ve been busy with creating Kubernetes `.yml` files for the openBalena containers. Because openBalena uses containers by default, I thought Kubernetes wouldn’t be a problem. But I was wrong. I’ve [forked the repository of openBalena](https://github.com/bartversluijs/open-balena/tree/k8s-support) to implement Kubernetes. I’ve added `scripts/k8s` and the `k8s` directory with the YAML files. I’ve not converted everything yet, because I ran into some problems. As it turns out, most containers use `systemd` to start one or more processes per container. But `systemd` isn’t well supported in containers. Why don’t the containers just run their program instead of using `systemd` to start it for them? I’ve googled to try and get `systemd` working in Kubernetes, but I didn’t succeed (yet). I’ve seen in an [issue in GitHub](https://github.com/balena-io/open-balena/issues/6), so I’m not the only one asking. And because that issue isn’t really active anymore, I thought maybe with the help of Balena and the community, we could reach a workable Kubernetes version of openBalena. It’d be awesome if we’d get it working, because scaling openBalena would be much simpler using Kubernetes. I hope by using the forum it can stir up the conversation about Kubernetes! ---
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 13, 2020, 7:34pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/4 "2020-06-13T19:34:03Z")
I’ve added more deployments, services, volumes and an Ingress to my fork. I got Redis, DB, S3, API, Registry & VPN running and connected with an Ingress. I’ve tested `balena login`, and it worked 🙂 I haven’t tested adding a device yet, but it probably doesn’t work because of the VPN configuration. I have only tested it on Docker Desktop for Mac. I’m going to try it out on DigitalOcean asap. It’d be awesome if I got it working and scaling is as easy as changing the replica’s. I’ll let you know about my progress, and it’d be nice if the Balena team could help testing it out! ---
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 14, 2020, 10:01am UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/5 "2020-06-14T10:01:12Z")
I’ve tested creating an app and provisioning a device to the k8s cluster on DigitalOcean. This works as expected, so that’s another milestone! I’ve added an nginx ingress to the cluster to handle the different subdomains and added a `cert-manager` manually. I’ll add this, after some testing, to the repository later on. But my next obstacle is the VPN. I haven’t managed to get it to work. I’ve added a static NodePort to the VPN service (`30001`) and added it to the LoadBalancer on DigitalOcean. When I go to: ```auto vpn.mydomain.io ``` I’m redirected to HTTPS and get a 502 error. I have another server running openBalena on Docker, and that responds a 503 error (which is probably the same for NGINX as it is for Haproxy). When I go to port `3128` via HTTP, I get a `method not allowed` on both servers, so this seems like it’s working. When I look into the logs of the device, I get the following output: ```auto Jun 14 09:58:15 20a9eed openvpn[2923]: Sun Jun 14 09:58:15 2020 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts Jun 14 09:58:15 20a9eed openvpn[2923]: Sun Jun 14 09:58:15 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]64.225.83.185:443 Jun 14 09:58:15 20a9eed openvpn[2923]: Sun Jun 14 09:58:15 2020 Socket Buffers: R=[87380->87380] S=[16384->16384] Jun 14 09:58:15 20a9eed openvpn[2923]: Sun Jun 14 09:58:15 2020 Attempting to establish TCP connection with [AF_INET]64.225.83.185:443 [nonblock] Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 TCP connection established with [AF_INET]64.225.83.185:443 Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 TCP_CLIENT link local: (not bound) Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 TCP_CLIENT link remote: [AF_INET]64.225.83.185:443 Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 WARNING: Bad encapsulated packet length from peer (18516), which must be > 0 and <= 1627 -- please ensure that --tun-mtu or --link-mtu is equal on both peers -- this condition could also indicate a possibl> Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 Connection reset, restarting [0] Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 SIGUSR1[soft,connection-reset] received, process restarting Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 Restart pause, 120 second(s) ``` It’s probably some misconfiguring on my part. But after looking in the `haproxy.cfg`, I don’t know what goes wrong. I’ve also never worked with haproxy before, so I don’t really understand the config. I hope you guys can help me out here, so I can make the k8s a success with openBalena! **Added question** _Is it possible to use S3 storage instead of the `open-balena-s3` with minio configured? And if so, is it also possible to use DigitalOcean Spaces (which is compatible with S3, AFAIK). This way, I can utilize all the nice things of openBalena and use the power of third-party services 🙂_ Got this working! Had to change the following environment variables in `config/activate`: ```auto - OPENBALENA_S3_ACCESS_KEY - OPENBALENA_S3_SECRET_KEY - OPENBALENA_S3_ENDPOINT - OPENBALENA_S3_BUCKETS - OPENBALENA_REGISTRY2_S3_BUCKET ``` Linked this to the DigitalOcean Spaces configuration I’ve got and everything of the Registry is saved to the right space. So I don’t need the S3 Minio container anymore and all is handled by a third-party S3 solution, which is much cheaper and convenient 🙂 ---
**Author:** ![dfunckt](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/dfunckt/32/56_2.png) [@dfunckt](https://forums.balena.io/u/dfunckt)\ **Post date:** [June 15, 2020, 11:37am UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/6 "2020-06-15T11:37:26Z")
Great stuff @bversluijs! You don’t need haproxy if you have other ways to route traffic into the containers – nginx should work just fine instead. I don’t think the VPN listens on 30001, but I’ll ask the team for pointers. ---
**Author:** ![wrboyce](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/wrboyce/32/374_2.png) [@wrboyce](https://forums.balena.io/u/wrboyce)\ **Post date:** [June 15, 2020, 11:47am UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/7 "2020-06-15T11:47:44Z")
Hi @bversluijs, would you be able to share the relevant sections of your nginx config for proxying connections to openvpn? Thanks. ---
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 15, 2020, 11:49am UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/8 "2020-06-15T11:49:21Z")
Hi @dfunckt, Thanks! And no, the VPN exposes port 80, 443 and 3128. I’ve added a NodePort in Kubernetes that forwards 3128 to 30001 (because NodePort has to be between 30000 and 32767). And the LoadBalancer proxies 3128 to 30001. Little bit weird, but that’s the way K8s and NodePort works. Only issue is, the docker container VPN exposes port 80, 443 and 3128. So I’m very curious which port does what. 3128 probably does the VPN-stuff, but 80 and 443 are HTTP and HTTP(S). But which one is needed? Because Ingress can only forward 1 port by default, which is 80 now. It forwards to HTTPS of course, but Ingress is set to 80. ---
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 15, 2020, 11:54am UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/9 "2020-06-15T11:54:05Z")
Hi @wrboyce, I’m using Ingress to forward everything. And I’m using the [nginx Ingress Controller](https://github.com/kubernetes/ingress-nginx). So I can’t create custom configs for nginx. The relevant configs are these files: [VPN expose service](https://github.com/bartversluijs/open-balena/blob/k8s-support/k8s/openbalena-vpn.service.yml) [Ingress config](https://github.com/bartversluijs/open-balena/blob/k8s-support/k8s/ingress.yml) I’m happy to explain it any other way, like via a chat, and debug the problem. This is the only piece I’m stuck at currently, so it would be very nice if we can debug the issue and I can use openBalena with Kubernetes. I’m currently working on that, so any help would be very appreciated! ---
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 15, 2020, 4:51pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/10 "2020-06-15T16:51:07Z")
I’ve tried installing HAProxy, because I keep getting errors with the VPN when using NGINX. But I couldn’t get HAProxy to work. So my only problem is the VPN at the moment which makes the openbalena unusable right now… Some more details: When I go to my docker openBalena instance to the IP:443, it shows an HTTP\_EMPTY\_RESPONSE error. When I go to my k8s openBalena instance, it shows `400 Bad Request: The plain HTTP request was sent to HTTPS port`. Any help would be appreciated! Even knowing what nginx configuration could be used could be a huge help! ---
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 16, 2020, 2:53pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/16 "2020-06-16T14:53:23Z")
I’ve tried everything using the nginx Ingress controller, but I’ve changed to the HAProxy Ingress controller. This seems to have more support for this. I’ve also created a [issue on the HAProxy Ingress controller repo](https://github.com/jcmoraisjr/haproxy-ingress/issues/602), because I can’t really accomplish what I’d like. It creates the following rules in the config: ```auto listen tcp-443 bind *:443 mode tcp server 10.244.0.92:443 10.244.0.92:443 check port 443 inter 2s send-proxy-v2 ``` But it looks like it’s loadbalancing between HTTPS and TCP, because the TLS check isn’t created. Any thoughts from your side would be very much appreciated. **Added question** Can someone explain where port 80 en 3128 is used for in the VPN container? And also, is an SSL certificate needed for `vpn.[mydomain].com`? Because the other solution is adding another loadbalancer for the `vpn.[mydomain].com` which only uses TCP instead of HTTP(S) with Ingress. But adding an SSL certificate for that domain is much more difficult. I’ve tried using a loadbalancer, that works. But the VPN is losing connection after a few minutes and tries to reconnect again. So it’s not very stable. I’ve exposed port 80 443 and 3128 via TCP to the LoadBalancer, so it’s weird that it loses connection all the time… ---
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 17, 2020, 11:12am UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/19 "2020-06-17T11:12:20Z")
I’ve been trying to get Kubernetes working with openBalena, but it’s simply not possible without some changes. I’ve created an [issue for the API](https://github.com/balena-io/open-balena-api/issues/379) for example. Also, it’s really hard/impossible to get the VPN working, because it’s on port 443. This is used by HTTPS for Ingress. The only other way is to add another LoadBalancer for `vpn.mydomain.com`. But I’m not sure if this is possible, because you lose HTTPS on `vpn.mydomain.com`. So without any debug help from the Balena team and added information, I’m giving up on getting it to work with Kubernetes, which is real unfortunate. Because it’s perfect for scaling. ---
**Author:** ![richbayliss](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/richbayliss/32/1054_2.png) [@richbayliss](https://forums.balena.io/u/richbayliss)\ **Post date:** [June 17, 2020, 12:18pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/20 "2020-06-17T12:18:42Z")
Hey, A couple of things here; firstly, thank you for trying to get this working. It has been on my to-do list for a long time but unfortunately it just doesn’t go high enough up the priority list. Whilst I am a maintainer on openBalena I also have other projects I am involved with that are taking my time. The second thing is that the VPN port can be changed. It’s an environment variable in the compose file and as long as you match it in the ingress side and in both the API and VPN service definitions, then it should work OK. The device pulls it’s VPN config from the API on startup so it should be happy to use any port it’s supplied. ---
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 17, 2020, 12:27pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/22 "2020-06-17T12:27:27Z")
Hi, Thanks for your reply. I understand you’re busy with other stuff, what Balena is doing is awesome, so no hard feelings there 🙂. But I’ve dedicated the last few days and weekend to get openBalena working with Kubernetes, because of the scaling advantages and I’d like to have a stable and scalable production environment for my company to use. Regarding the VPN, you’ve probably decided to run the VPN on port 443 because of firewall issues (correct me if I’m wrong). Every company has this port enabled for outgoing packets, if they’re in their right mind. So I’m not a really big fan of changing this port. Another LoadBalancer is no issue here, but it’s impossible to run the VPN’s HTTP(S) via LoadBalancer #1 and VPN TCP (443) via LoadBalancer #2, because of DNS issues (assigning 1 IP address to the `vpn.domain.com`). The question is, if this is needed or not. Do you need HTTPS to run on that IP address for `vpn.domain.com`, or is it only used for the VPN? And probably an SSL certificate isn’t needed too? ---
**Author:** ![richbayliss](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/richbayliss/32/1054_2.png) [@richbayliss](https://forums.balena.io/u/richbayliss)\ **Post date:** [June 17, 2020, 12:41pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/23 "2020-06-17T12:41:20Z")
So port 443 is used for this very reason; as you say all corporate firewall allow traffic down that port. There is an API within the VPN but it’s not listening on 443. You can assume all traffic to the VPN service on 443 is OpenVPN. You can also run the `vpn.domain.com` on a different IP – we use the FQDN in all places so it’ll be resolved when needed. As I haven’t done this before, I might have missed something, but I don’t think it should be an issue. ---
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 17, 2020, 1:04pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/25 "2020-06-17T13:04:55Z")
Thanks for your confirmation about the port and IP-related complications! As you’ve already seen, I’ll have to wait until the API issue is fixed (and better yet, [this k8s issue](https://github.com/kubernetes/kubernetes/issues/66607) is fixed), before I can finish the k8s configuration. Thought this was a simple k8s project, because openbalena uses container, but I couldn’t be more wrong 😅. I’ve learned much about openbalena and k8s while trying to get this working with Kubernetes. So I got that going for me, which is nice 🙃. To be continued… ---
**Author:** ![richbayliss](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/richbayliss/32/1054_2.png) [@richbayliss](https://forums.balena.io/u/richbayliss)\ **Post date:** [June 17, 2020, 1:12pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/26 "2020-06-17T13:12:23Z")
the middleware checks to make sure that the `X-Forwarded-Proto` et-al headers are on the request and if not, redirects to HTTPS – `res.redirect(301, `https://{API\_HOST}{req.url}`);` Could you have this header injected somehow? Also, do you have the manifests publicly hosted that I could take a look at? ---
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 17, 2020, 1:27pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/28 "2020-06-17T13:27:14Z")
Hmm, I’m thinking of a solution to add this header on the request when the communication is internal. I don’t think, other than changing the `open-balena` repo’s with this added header based on an ENV variable, to do this in Kubernetes. But I’m open for suggestions! And yeah, the manifests are located [here](https://github.com/bartversluijs/open-balena/tree/k8s-support/k8s). Feel free to change them! I’m also available to chat with you/your team in Discord/Slack or another app. Thank you for taking time to look into this! ---
**Author:** ![chriswiggins](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/chriswiggins/32/5131_2.png) [@chriswiggins](https://forums.balena.io/u/chriswiggins)\ **Post date:** [June 17, 2020, 8:21pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/29 "2020-06-17T20:21:08Z")
Just following along here - is there a reason you’re trying to use ingress for the VPN? Why not just use a LoadBalancer service, with externalTrafficPolicy: Local so that source IPs are preserved? You’d need two external IPs but that should be fine because of DNS? ---
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 17, 2020, 8:44pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/30 "2020-06-17T20:44:13Z")
Hi, That’s no problem at all! The reason I was trying to use Ingress is because open-balena is using 1 HAProxy by default, so I thought it could be done in 1 HAProxy Ingress. I’ve learned (the hard way) that it’s not possible to get it working stable. And because of the extra information provided by the @richbayliss, it’s not necessary to also run HTTPS for the VPN. So it’s definitely the best solution. But because open-balena uses communication over HTTPS to communicate to other containers, and there is an ‘issue’ in Kubernetes that prevents this, it’s not possible to get open-balena working with Kubernetes. If you’ve any suggestions, please let me know, because I’m keen to get open-balena working with Kubernetes! ---
**Author:** ![chriswiggins](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/chriswiggins/32/5131_2.png) [@chriswiggins](https://forums.balena.io/u/chriswiggins)\ **Post date:** [June 17, 2020, 8:50pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/31 "2020-06-17T20:50:32Z")
> [@bversluijs](#): > > But because open-balena uses communication over HTTPS to communicate to other containers, and there is an ‘issue’ in Kubernetes that prevents this, it’s not possible to get open-balena working with Kubernetes. If you’ve any suggestions, please let me know, because I’m keen to get open-balena working with Kubernetes! Ah ok I think I see where you’re hitting that issue. Thats a hard one, unless you get the containers to talk to each other on internal kubernetes DNS addresses, rather than their public addresses? Or do you still hit the TLS issue? ---
**Author:** ![bversluijs](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/bversluijs/32/3954_2.png) [@bversluijs](https://forums.balena.io/u/bversluijs)\ **Post date:** [June 17, 2020, 8:56pm UTC](https://forums.balena.io/t/kubernetes-for-scaling/132102/32 "2020-06-17T20:56:34Z")
Every container talks to each other using HTTPS, so that’s a hard one. I’ve created [an issue](https://github.com/balena-io/open-balena-api/issues/379), which will fix the automatic redirect. But as you can see [here](https://github.com/balena-io/open-balena-vpn/search?q=BALENA_API_HOST&unscoped_q=BALENA_API_HOST), the VPN uses forced `https://` before the `BALENA_API_HOST` environment variable, so it’s impossible to fix this without creating my own containers, which isn’t optimal… So the best solution is to fix a workaround for the Kubernetes issue. [Next page](https://forums.balena.io/t/kubernetes-for-scaling/132102.md?page=2)