I am running an experimental cluster with some devices running balenaOS connected to my balenaCloud application:
2x rasp pi 3B
1x rasp pi 4
My goal is for them to run a peer-to-peer cluster, and discover each other with the assistance of the resin VPN interface that balenaOS created/manages for me. I am able to discover the exact address of any particular device, but there is no docs I can find which speak to how the various peers of a single application on balenaCloud are intended to address each other.
It seems like dnsmasq is employed on the host OS and may provide resolution to application containers, but i am unsure how to proceed with peer discovery using it, what names to attempt to resolve, etc.
Is there a service discovery component provided to resin VPN clients? is the dnsmasq able to resolve my application devices by their device name in balena console somehow? Is there a DNS resolution pattern that I can use to discover other peers that are online ?
Hi @eve an interesting project. Surprisingly I have not heard of anyone doing that before so we have forwarded the question to an engineer who might have.
The balena API allows you to retrieve all devices for an application, see the documentation here: https://www.balena.io/docs/reference/api/resources/device/#devices-by-application .
So the remaining question is - is there is a way for a device to resolve the uuid of another device to a valid VPN-IP and contact that IP via the VPN.
Using public device URL’s that should be possible but it might be a bit of a detour compared to direct IP addressing via the VPN. Direct communication on the VPN on the other hand might be restricted by topology or firewall rules as it might not be desirable to allow arbitrary connections between devices on the VPN.
I will update you once we find reliable information on this topic.
@eve we found this https://www.balena.io/docs/learn/welcome/security/#runtime-management statement that basically states what I suspected. Device to device communication via VPN is actively blocked to keep compromised devices from spreading their infestation through the VPN.
One of our engineers suggested, that the cleanest solution would probably be to set up your own VPN - clearly that will be the fastest solution as you are not sharing the resources with huge amounts of other devices and you can set up device discovery using appropriate mechanisms.
Otherwise public device URLs might be the way to go.
Thanks @samothx for this information. It sounds like Public Device URL will allow my nodes to contact each other to federate OpenVPN between them, and then pass information through the tunnel.
Two questions that would be important for this flow to work:
How does TLS termination work with public device URLs? If I were to set up a CNAME record for one of these devices, say for pi1.example.com, and I provided my own PKI to connecting VPN clients, how would I provide my device with my custom private key and certificate to serve pi1.eve.com so they are trusted?
is there any way to discover what the device unique ID is or will be ? Suppose I have this all set up and add a 4th device, and I want it to discover the other nodes which are already running – is there a way for it to learn (perhaps by connecting to Balena) what other device IDs exist in my application?
TLS is terminated at our proxy server’s ELB, from there a plaintext request hits our proxy server which is then tunneled via the vpn to the device. So the TLS portion of the HTTPS request is terminated at our load balancers, but the request is re-encrypted between our proxy servers and the device.
As for detecting the UUID of new devices, you can certainly use our api for this, most likely the following endpoint would be of use: https://www.balena.io/docs/reference/api/resources/device/#devices-by-application and you could either use the createdAt fiield, or examine the list of known devices against the list returned, to detect any newcomers.
Let me know if you need any further information.
EDIT: to be more clear, your device would serve plain TCP connections.
That clarifies everything I need to know about the TLS situation. Now, regarding the API you referred to, I read that page https://www.balena.io/docs/learn/manage/account/#api-keys which maybe suggests that you can produce with a business account, an API key for someone with “observer” role which is therefore limited in scope. But is it possible to produce a token with more granular levels of access, or for example, to issue myself an API key which is for myself as a user but only having some of the “scopes” (in oauth sense) that apply to me & my resource access?
In other words, I would prefer not to create a new “member” because indeed the devices are more accurate to authenticate as a “limited agent on my behalf”.
But is it possible to produce a token with more granular levels of access, or for example, to issue myself an API key which is for myself as a user but only having some of the “scopes” (in oauth sense) that apply to me & my resource access?
Unfortunately it is not currently possible to produce a token with a more granular level of access, but it is a feature that is on our radar to be implemented in the future, so that admin users can create additional accounts and keys with specified access permissions to listed resources and platform features.