Kubernetes for scaling

openBalena
1

Hi all,

I’ve been busy with creating Kubernetes .yml files for the openBalena containers. Because openBalena uses containers by default, I thought Kubernetes wouldn’t be a problem. But I was wrong.

I’ve forked the repository of openBalena to implement Kubernetes. I’ve added scripts/k8s and the k8s directory with the YAML files. I’ve not converted everything yet, because I ran into some problems.

As it turns out, most containers use systemd to start one or more processes per container. But systemd isn’t well supported in containers. Why don’t the containers just run their program instead of using systemd to start it for them? I’ve googled to try and get systemd working in Kubernetes, but I didn’t succeed (yet).

I’ve seen in an issue in GitHub, so I’m not the only one asking. And because that issue isn’t really active anymore, I thought maybe with the help of Balena and the community, we could reach a workable Kubernetes version of openBalena. It’d be awesome if we’d get it working, because scaling openBalena would be much simpler using Kubernetes.

I hope by using the forum it can stir up the conversation about Kubernetes!

2 Likes
4

I’ve added more deployments, services, volumes and an Ingress to my fork. I got Redis, DB, S3, API, Registry & VPN running and connected with an Ingress. I’ve tested balena login, and it worked :slight_smile:

I haven’t tested adding a device yet, but it probably doesn’t work because of the VPN configuration. I have only tested it on Docker Desktop for Mac. I’m going to try it out on DigitalOcean asap. It’d be awesome if I got it working and scaling is as easy as changing the replica’s.

I’ll let you know about my progress, and it’d be nice if the Balena team could help testing it out!

5

I’ve tested creating an app and provisioning a device to the k8s cluster on DigitalOcean. This works as expected, so that’s another milestone!

I’ve added an nginx ingress to the cluster to handle the different subdomains and added a cert-manager manually. I’ll add this, after some testing, to the repository later on. But my next obstacle is the VPN. I haven’t managed to get it to work. I’ve added a static NodePort to the VPN service (30001) and added it to the LoadBalancer on DigitalOcean. When I go to:

vpn.mydomain.io

I’m redirected to HTTPS and get a 502 error. I have another server running openBalena on Docker, and that responds a 503 error (which is probably the same for NGINX as it is for Haproxy). When I go to port 3128 via HTTP, I get a method not allowed on both servers, so this seems like it’s working.

When I look into the logs of the device, I get the following output:

Jun 14 09:58:15 20a9eed openvpn[2923]: Sun Jun 14 09:58:15 2020 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Jun 14 09:58:15 20a9eed openvpn[2923]: Sun Jun 14 09:58:15 2020 TCP/UDP: Preserving recently used remote address: [AF_INET]64.225.83.185:443
Jun 14 09:58:15 20a9eed openvpn[2923]: Sun Jun 14 09:58:15 2020 Socket Buffers: R=[87380->87380] S=[16384->16384]
Jun 14 09:58:15 20a9eed openvpn[2923]: Sun Jun 14 09:58:15 2020 Attempting to establish TCP connection with [AF_INET]64.225.83.185:443 [nonblock]
Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 TCP connection established with [AF_INET]64.225.83.185:443
Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 TCP_CLIENT link local: (not bound)
Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 TCP_CLIENT link remote: [AF_INET]64.225.83.185:443
Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 WARNING: Bad encapsulated packet length from peer (18516), which must be > 0 and <= 1627 -- please ensure that --tun-mtu or --link-mtu is equal on both peers -- this condition could also indicate a possibl>
Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 Connection reset, restarting [0]
Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 SIGUSR1[soft,connection-reset] received, process restarting
Jun 14 09:58:16 20a9eed openvpn[2923]: Sun Jun 14 09:58:16 2020 Restart pause, 120 second(s)

It’s probably some misconfiguring on my part. But after looking in the haproxy.cfg, I don’t know what goes wrong. I’ve also never worked with haproxy before, so I don’t really understand the config.

I hope you guys can help me out here, so I can make the k8s a success with openBalena!

Added question
Is it possible to use S3 storage instead of the open-balena-s3 with minio configured? And if so, is it also possible to use DigitalOcean Spaces (which is compatible with S3, AFAIK). This way, I can utilize all the nice things of openBalena and use the power of third-party services :slight_smile:
Got this working! Had to change the following environment variables in config/activate:

- OPENBALENA_S3_ACCESS_KEY
- OPENBALENA_S3_SECRET_KEY
- OPENBALENA_S3_ENDPOINT
- OPENBALENA_S3_BUCKETS
- OPENBALENA_REGISTRY2_S3_BUCKET

Linked this to the DigitalOcean Spaces configuration I’ve got and everything of the Registry is saved to the right space. So I don’t need the S3 Minio container anymore and all is handled by a third-party S3 solution, which is much cheaper and convenient :slight_smile:

6

Great stuff @bversluijs! You don’t need haproxy if you have other ways to route traffic into the containers – nginx should work just fine instead. I don’t think the VPN listens on 30001, but I’ll ask the team for pointers.

7

Hi @bversluijs, would you be able to share the relevant sections of your nginx config for proxying connections to openvpn? Thanks.

8

Hi @dfunckt,

Thanks! And no, the VPN exposes port 80, 443 and 3128. I’ve added a NodePort in Kubernetes that forwards 3128 to 30001 (because NodePort has to be between 30000 and 32767). And the LoadBalancer proxies 3128 to 30001. Little bit weird, but that’s the way K8s and NodePort works.

Only issue is, the docker container VPN exposes port 80, 443 and 3128. So I’m very curious which port does what. 3128 probably does the VPN-stuff, but 80 and 443 are HTTP and HTTP(S). But which one is needed? Because Ingress can only forward 1 port by default, which is 80 now. It forwards to HTTPS of course, but Ingress is set to 80.

9

Hi @wrboyce,

I’m using Ingress to forward everything. And I’m using the nginx Ingress Controller. So I can’t create custom configs for nginx.

The relevant configs are these files:
VPN expose service
Ingress config

I’m happy to explain it any other way, like via a chat, and debug the problem. This is the only piece I’m stuck at currently, so it would be very nice if we can debug the issue and I can use openBalena with Kubernetes. I’m currently working on that, so any help would be very appreciated!

10

I’ve tried installing HAProxy, because I keep getting errors with the VPN when using NGINX. But I couldn’t get HAProxy to work. So my only problem is the VPN at the moment which makes the openbalena unusable right now…

Some more details:
When I go to my docker openBalena instance to the IP:443, it shows an HTTP_EMPTY_RESPONSE error. When I go to my k8s openBalena instance, it shows 400 Bad Request: The plain HTTP request was sent to HTTPS port.

Any help would be appreciated! Even knowing what nginx configuration could be used could be a huge help!

16

I’ve tried everything using the nginx Ingress controller, but I’ve changed to the HAProxy Ingress controller. This seems to have more support for this.

I’ve also created a issue on the HAProxy Ingress controller repo, because I can’t really accomplish what I’d like. It creates the following rules in the config:

listen tcp-443
    bind *:443
    mode tcp
    server 10.244.0.92:443 10.244.0.92:443 check port 443 inter 2s send-proxy-v2

But it looks like it’s loadbalancing between HTTPS and TCP, because the TLS check isn’t created.
Any thoughts from your side would be very much appreciated.

Added question
Can someone explain where port 80 en 3128 is used for in the VPN container?
And also, is an SSL certificate needed for vpn.[mydomain].com? Because the other solution is adding another loadbalancer for the vpn.[mydomain].com which only uses TCP instead of HTTP(S) with Ingress. But adding an SSL certificate for that domain is much more difficult.

I’ve tried using a loadbalancer, that works. But the VPN is losing connection after a few minutes and tries to reconnect again. So it’s not very stable. I’ve exposed port 80 443 and 3128 via TCP to the LoadBalancer, so it’s weird that it loses connection all the time…

19

I’ve been trying to get Kubernetes working with openBalena, but it’s simply not possible without some changes. I’ve created an issue for the API for example. Also, it’s really hard/impossible to get the VPN working, because it’s on port 443. This is used by HTTPS for Ingress. The only other way is to add another LoadBalancer for vpn.mydomain.com. But I’m not sure if this is possible, because you lose HTTPS on vpn.mydomain.com.

So without any debug help from the Balena team and added information, I’m giving up on getting it to work with Kubernetes, which is real unfortunate. Because it’s perfect for scaling.

20

Hey,

A couple of things here; firstly, thank you for trying to get this working. It has been on my to-do list for a long time but unfortunately it just doesn’t go high enough up the priority list. Whilst I am a maintainer on openBalena I also have other projects I am involved with that are taking my time.

The second thing is that the VPN port can be changed. It’s an environment variable in the compose file and as long as you match it in the ingress side and in both the API and VPN service definitions, then it should work OK. The device pulls it’s VPN config from the API on startup so it should be happy to use any port it’s supplied.

22

Hi,

Thanks for your reply. I understand you’re busy with other stuff, what Balena is doing is awesome, so no hard feelings there :slight_smile:.
But I’ve dedicated the last few days and weekend to get openBalena working with Kubernetes, because of the scaling advantages and I’d like to have a stable and scalable production environment for my company to use.

Regarding the VPN, you’ve probably decided to run the VPN on port 443 because of firewall issues (correct me if I’m wrong). Every company has this port enabled for outgoing packets, if they’re in their right mind. So I’m not a really big fan of changing this port. Another LoadBalancer is no issue here, but it’s impossible to run the VPN’s HTTP(S) via LoadBalancer #1 and VPN TCP (443) via LoadBalancer #2, because of DNS issues (assigning 1 IP address to the vpn.domain.com). The question is, if this is needed or not. Do you need HTTPS to run on that IP address for vpn.domain.com, or is it only used for the VPN? And probably an SSL certificate isn’t needed too?

23

So port 443 is used for this very reason; as you say all corporate firewall allow traffic down that port.

There is an API within the VPN but it’s not listening on 443. You can assume all traffic to the VPN service on 443 is OpenVPN. You can also run the vpn.domain.com on a different IP – we use the FQDN in all places so it’ll be resolved when needed. As I haven’t done this before, I might have missed something, but I don’t think it should be an issue.

1 Like
25

Thanks for your confirmation about the port and IP-related complications!
As you’ve already seen, I’ll have to wait until the API issue is fixed (and better yet, this k8s issue is fixed), before I can finish the k8s configuration. Thought this was a simple k8s project, because openbalena uses container, but I couldn’t be more wrong :sweat_smile:.

I’ve learned much about openbalena and k8s while trying to get this working with Kubernetes. So I got that going for me, which is nice :upside_down_face:.

To be continued…

26

the middleware checks to make sure that the X-Forwarded-Proto et-al headers are on the request and if not, redirects to HTTPS – res.redirect(301, https://{API_HOST}{req.url});

Could you have this header injected somehow?

Also, do you have the manifests publicly hosted that I could take a look at?

28

Hmm, I’m thinking of a solution to add this header on the request when the communication is internal. I don’t think, other than changing the open-balena repo’s with this added header based on an ENV variable, to do this in Kubernetes. But I’m open for suggestions!

And yeah, the manifests are located here. Feel free to change them! I’m also available to chat with you/your team in Discord/Slack or another app.

Thank you for taking time to look into this!

29

Just following along here - is there a reason you’re trying to use ingress for the VPN?

Why not just use a LoadBalancer service, with externalTrafficPolicy: Local so that source IPs are preserved?

You’d need two external IPs but that should be fine because of DNS?

30

Hi,

That’s no problem at all! The reason I was trying to use Ingress is because open-balena is using 1 HAProxy by default, so I thought it could be done in 1 HAProxy Ingress. I’ve learned (the hard way) that it’s not possible to get it working stable.

And because of the extra information provided by the @richbayliss, it’s not necessary to also run HTTPS for the VPN. So it’s definitely the best solution.

But because open-balena uses communication over HTTPS to communicate to other containers, and there is an ‘issue’ in Kubernetes that prevents this, it’s not possible to get open-balena working with Kubernetes. If you’ve any suggestions, please let me know, because I’m keen to get open-balena working with Kubernetes!

31

Ah ok I think I see where you’re hitting that issue. Thats a hard one, unless you get the containers to talk to each other on internal kubernetes DNS addresses, rather than their public addresses? Or do you still hit the TLS issue?

32

Every container talks to each other using HTTPS, so that’s a hard one.
I’ve created an issue, which will fix the automatic redirect. But as you can see here, the VPN uses forced https:// before the BALENA_API_HOST environment variable, so it’s impossible to fix this without creating my own containers, which isn’t optimal…

So the best solution is to fix a workaround for the Kubernetes issue.