I wanted to let you know of another side project that would be great to get some feedback on, and I think will be useful for many of you.
Balena Virt has had a makeover to include an option to try the Balena Cloud without the need for hardware, flashing etc. (no need to leave your desk) and to speed up development flows by starting development on powerful remote systems (like Digital Ocean Droplets) running a virtualised balena OS. You can access it on your network just like any real device.
There are now a number of options for using Balena Virt:
Turning a single Intel NUC into a small fleet of devices for testing and development
Running on a Digital Ocean Droplet, providing an easy way to try Balena Cloud without the need for physical hardware, and to provide a powerful development platform
Running on other VPS services and hardware. See the advanced setup which provides containers that can be used and steps for using Tailscale for multi-device setups.
In particular, the DigitalOcean option I hope will prove useful, it enables a Digital Ocean Droplet to run balena OS and for you to use it just like any other piece of hardware on your network (balena push, balena join etc.).
I have not played with any of the proposed systems, but this is an excellent idea in principle. It is a great way to have a testing / development without breaking production or first requiring hardware.
For my personal use cases, I would ideally run a local VM running BalenaOS and do my development work locally.
I’ve had a play and have the following containers running locally on a docker install:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
f7ff38e4c91b ghcr.io/balena-labs-research/balena-virt:latest "/app/start.sh" 59 minutes ago Up 59 minutes suspicious_almeida
407db4c61283 ghcr.io/balena-labs-research/balena-virt-networking:latest "/app/start.sh" About an hour ago Up About an hour distracted_lehmann
Is there no easy access akin to balena ssh ? I wonder too if there’s a way to add them as virtual devices to the Balena Cloud, so I can see all fleets in 1 location. More reading and testing to do I suppose. Great work
Great to hear you have had a chance to spin it up.
The containers should start up locally on your system, and if you are running Ubuntu and an Intel you may even get good performance. The problem is, it replies on KVM. To run this locally on Mac or Windows it involves a different process. You could still get them to start, but they will likely be very slow.
I did look for a bit at some other operating build processes, but they all needed some configuration on the main system, and added a lot more to have to support.
Is there no easy access akin to balena ssh ?
Assuming they have started on your local system, you might be able to see them via balena scan. Or if you are able to get the IP address assigned, which could probably be found in the balena-virt-networking container logs, you could try ssh the.ip.address -p 22222). You mind find though that it hasn’t started, it is very slow without hardware acceleration and could just be stuck in a boot process.
Fortunately for me I’m using amd64 Debian with KVM.
Assuming they have started on your local system, you might be able to see them via balena scan. Or if you are able to get the IP address assigned, which could probably be found in the balena-virt-networking container logs, you could try ssh the.ip.address -p 22222). You mind find though that it hasn’t started, it is very slow without hardware acceleration and could just be stuck in a boot process.
Found it and logged in:
~/c/g/balena> docker logs 407db4c61283
...
dnsmasq-dhcp: DHCP, IP range 10.0.3.10 -- 10.0.3.110, lease time 1h
...
~/c/g/balena> balena ssh 10.0.3.10
Last login: Tue Jan 3 17:29:25 2023
root@0498005:~#
Okay, so this is very cool. Last part for me, I’m so used to using the Balena Cloud service to deploy services that I’m stuck in my thinking (without testing or reading anything) as to how to deploy a service to this virtual device. What’s the intended solution to this? Beyond the basics I’m not very experienced with Balena development, so thanks in advance! I should test first before asking unnecessary questions; a quick reread of your original post even says about balena push etc. Will have another play.
I may have missed a network step, here is an attempt to test build BalenaSound:
~/c/g/b/balena-sound (master)> balena push 10.0.3.10
[Info] Starting build on device 10.0.3.10
[Build] [spotify] Step 1/19 : ARG BALENA_ARCH=amd64
[Build] [spotify] Step 2/19 : FROM balenalib/$BALENA_ARCH-alpine:3.16 as librespot-builder
[Build] [audio] Step 1/6 : FROM bh.cr/balenalabs/audio-amd64
[Build] [multiroom-server] Step 1/9 : FROM bh.cr/g_tomas_migone1/multiroom-amd64
[Build] [hostname] Step 1/3 : FROM bh.cr/g_tomas_migone1/hostname-amd64
[Build] [airplay] Step 1/11 : FROM mikebrady/shairport-sync:3.3.8 as shairport
[Build] [multiroom-client] Step 1/8 : FROM bh.cr/g_tomas_migone1/multiroom-amd64
[Build] [bluetooth] Step 1/6 : FROM bh.cr/balenalabs/bluetooth-amd64
[Build] [sound-supervisor] Step 1/10 : FROM balenalib/amd64-node:14 as build
Some services failed to build:
audio: Get "https://bh.cr/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
sound-supervisor: Get "https://registry-1.docker.io/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
multiroom-server: Get "https://registry-proxy.balena-cloud.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
multiroom-client: Get "https://registry-proxy.balena-cloud.com/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
bluetooth: Get "https://registry-proxy.balena-cloud.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
airplay: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
spotify: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
hostname: Get "https://registry-proxy.balena-cloud.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Strange that there is network timeouts. Can you ping google.com from inside the VM? Is the network container running with --network host? It should create a network interface that is used for traffic, which seems like it is working as you can get inside. It can be a little finicky though, which is one of the reasons we opted for Digital Ocean where we can keep the system type stable. Happy to dig a little with you though, it’s all good info.
Yes, so this was actually my error. I saw an IP address in the network log without reading the line correctly; even posted it above, and took the first address of the DHCP range as the balena containers IP, it obviously did not work! I then noticed the released DHCP address further down the log and was then able to successfully deploy balenasound to it; I had no way to test the actual sound part, but the container became available within the spotify client. So this is really cool!
Last bit then, which I may have overlooked, how would I have this virtual device show up in balena cloud?
I don’t think we will be working on any of these issues for a while as they are not breaking problems, just some nice features. If you have some time to take a look though, would love some help with this project.
