Yes! It’s almost there @dtischler - right now the backend endpoints are all functional for adding/removing/editing device variables. You can test it out with the postman collection if you would like to, or hit the endpoints directly. It should be added to the UI soon though!
2 Likes
Hi guys, this question is a wifi expert on the Balena team.
Do you know of any successful implementations of “hot swapping” wifi networks on a device? We have a discussion going on repo for this project about how it would be nice to be able to change to another wifi network while already having a successful connection, but perhaps too complex to implement on constrained devices like a rpi.
Without hot swap a user is forced to go through the slightly more cumbersome forget wifi flow, which puts device into AP mode, required the user to connect to that devices own wifi, select a good network, go back to his own wifi to get back on the web, and confirm that the device connected.
The desired behavior would be similar to your PC. You change change networks at will without needing to do essentially a new setup every time.
@dtischler @mpous have you seen anyone pull this off in any other projects?
2 Likes
It’s a question of being able to refresh the list of available networks while already connected to a network. It has seemed like NetworkManager and IW have had trouble in the past doing this. Not sure how widely spread the issue it?
Do you know of any successful implementations of “hot swapping” wifi networks on a device?
You don’t need to “swap” wifi network connection profiles because the “key” of them is the WiFI SSID,
so all you need is to “add” new wifi network connection profiles for the different WiFI SSID you want
to be able to connect to.
Then, depending on the results of the WiFi network scan, NetworkManager activates the “best” WiFi connection profile on the WiFi adapter.
If a device finds multiple WiFi access points for which it has connection profiles, it will usually try to keep using the current access point, but over the NetworkManager DBus API, this can be influenced.: New WiFi connection profiles can be added, existing profiles can be updated.
Also over DBus, it is possible to send a request to NetworkManager to try to activate a new/different WiFi connection profile. This needs a connection management agent in a container (or forwarding the DBus socket of the Balena host OS using socat over TCP):
If the DBus socket is forwarded into a container on your own server, all of the DBus services on the device are reachable, including systemd and NetworkManager. systemctl and nmcli could be used for full management over the forwarded DBus.
A few further ideas about a connection management agent in a container on the device itself:
Ultimately, the Balena supervisor container would the final destination of such connection management agent, and it needs an API over which it can be controlled from the cloud.
Since there are many values to be transmitted (e.g. getting a status report which connections are active and their connectivity status and also the requests to create a new connection profile) it makes sense to serialize/deserialize all this information to/from JSON.
Some examples:
This example program can be used to define a new WiFi connection profile:
This JSON-RPC framework supports many transport layers and all what is needed is to implement the RPC methods for the server which translate JSON-RPC API calls to NetworkManager DBus API calls:
https://pjrpc.readthedocs.io/en/latest/
It would be nice if such RPC calls to the connection manager agent could be made over the supervisor API, but I don’t know if it is suited to get quick answers from the device. For now, it looks like a forwarded banena tunnel/ssh connection would work to make JSON-RPC calls.
1 Like
@bkaindl excellent info thank you! The access to d-bus with socat looks very interesting, that feels like the right path from the UI container. Will do some more digging in that direction.
Also @maggie0002 I recall you mentioning that throwing authentication on the local UI to protect config changes is a little tricky. Perhaps this helps? https://pocketbase.io/
Fascinating project that embed SQLite in a single binary, comes with OAuth2 out of the box (e.g. sign up/in with Google, Github, etc), realtime subscriptions, and a pretty slick admin UI.
I’ve been experimenting a bit and it’s trivial to set up and throw into a lightweight backend container.
1 Like
Thanks @barryjump! I was just talking about the device-ui authentication with @nucleardreamer yesterday, perfect timing, and pinging so he can take a look too.
1 Like
@barryjump The part of my answer with the link to the forwarding of the NetworkManager socket using socat was mostly about remote access to a NetworkManager socket (from a different machine).
For access to the local NetworkManager on a BalenaOS device from within a balena container, the easiest and best solution is simply using the complete description provided by Balena in their documentation for “Changing the Network at Runtime” (link below).
Put into code, you simply add the label and the environment variable below to the definition of your network-UI container in docker-compose.yml
network-ui-container:
build: ...
labels:
io.balena.features.dbus: '1'
environment:
- 'DBUS_SYSTEM_BUS_ADDRESS=unix:path=/host/run/dbus/system_bus_socket'
Just for reference, about the settings above:
- Documentation about the labels: Multiple containers - Balena Documentation
- System-DBus access in a container: Communicate outside the container - Balena Documentation
Example python script for adding a new Wifi connection, it is directly below the paragraph starting with:
The following example will add a new WiFi connection file to your host OS NetworkManager configuration in
/etc/NetworkManager/system-connections
Link: Network Setup on balenaOS - Balena Documentation
For advanced dynamic network communication, I can also recommend the python-sdbus-networkmanager library because it provides a higher-level, NetworkManager-specific API for the NetworkManager’s DBUS API, and has a blocking and and asyncio variant, while still giving access to every feature of the API, and in the examples, I provided an example standalone script for adding a WiFI connection (tested for WPA-PSK, other authentication methods are possible, but might need other parameters for authentication):
Besides having both a blocking and an asnycio API, it’s unique feature (especially for development) is that the parent library, python-sdbus, which provides the DBus layer can really connect to a remote DBus using openssh.
I use this for development: It means you create a Host entry in ~/.ssh/config which connects to a BalenaOS device (SSH on port 22222), and use that entry with sdbus.sd_bus_open_system_remote("balenaos-device") to develop a piece of code locally on your development machine (e.g. a Linux Notebook, Linux Server or Linux VM), but access the NetworkManager of a BalenaOS device over that SSH connection for development:
Host balena-device
Hostname 10.0.0.22
Port 22222
This even works if you need to use balena tunnel uuid -p 22222 to forward the BalenaOS SSH port 22222 using a tunnel. For this you start the balena tunnel command yourself on the shell and then, you use this entry in ~/.ssh/config to connect to the BalenaOS device:
Host local-balena-tunnel-on-22222
Hostname localhost
Port 22222
User root
Documentation on the corresponding python-sdbus function: Common API — python-sdbus documentation
@bkaindl, interesting approach there to your dev environment forward the DBUS socket. I have been toying around with different ideas for development processes, would be interested to hear more about your setup. What is it you are developing locally, Python? I had looked at an approach using an NFS server to allow files from a device to be mounted to the local system so development could be done locally but alter a real device. There is some great work done by @Mikhail on containerising NFS servers (Release Balena NFS 1.1.0 · VolkovLabs/balena-nfs · GitHub). It could allow things like hot reloading for UI development, although that did make things a little more complicated as the containers had to start in different ways.
I’m thinking opening the DBUS socket is an interesting approach. You mention developing locally in a Linux VM, presumably to keep the same architecture which I guess is a little more of a barrier?
There isn’t really any specific question or comment here, just to say that I’m really interested in brainstorming it a bit if you have any ideas, thoughts, musings etc. Maybe there isn’t going to be only one development environment solution, and need to start thinking about mapping different scenarios and build patterns and creating something for each? Although I’m still not quite ready to give up on trying to find a silver bullet solution.
The development environment is something I had looked at to integrate in to the starter UI, which is how these all link together in this thread.
1 Like
@maggie0002 Thank you for mentioning NFS project. NFS is working great for our projects and recently we added support for NFS4 introduced in the latest version of balenaOS.
Similar to the work you started with standardizing balena-cli action, I would love to see an UI project, which everybody can use and rely on. We built our own version of UI based on Grafana (GitHub - VolkovLabs/volkovlabs-balena-app: Balena Application plugin for @grafana.). Grafana is commonly used for Balena projects and provides authentication and other capabilities out of the box.
It’s available as an App and can be deployed to try on: balenaHub: an easier way to find and publish fleets, apps, and blocks for edge devices.
1 Like
Hi @Mikhail,
That is exactly what is in the pipeline! It started with the Balena Device UI (GitHub - maggie0002/balena-device-ui: A UI for Balena devices.) and the intent is now to evolve that quickly in to a modular “ui-builder”, where you can customise what components you would like and where, and set branding and colours all through one .yaml configuration file. Going to really put the foot on the gas in doing this and it has lots of scope for features.
It is also the plan to move it in to the “balena-community” org (balena-community · GitHub) where you found the GitHub Action (Community built Github Deploy and Preload action) and evolve it in the same way by getting as much scope from everyone as possible throughout the build process, and to make it a community project where we can be agile, move quickly, get lots of contributions and input and ensure it serves all the different needs of different projects. The balena-community org is new and experimental, and all going to be based and shaped on how people respond or interact with it (or don’t) to see if it is something that will continue. My feeling is it has a ton of scope to get us all together around these common goals.
In that vein, it would be fantastic to get as much input as possible on the UI, and can be raised as issues on the current repository (GitHub - maggie0002/balena-device-ui: A UI for Balena devices.) which will be moved over to the community org when the shift happens. This could be anything from ideas, use cases, feature requests, brainstorming, even a spec for your own use case would be useful. It’s not to say I or the other community people will work on all of the feature requests, but that mass data is so vital in shaping it towards being most useful, ensuring there is space for those features in the future and so forth. I should ping @keenanjohnson here who started this thread to encourage the same input on the shape of a UI.
Watch this space, and if anyone has other ideas on things we could all work on together that fills a real collective need then do mention them on the forums (like your NFS idea), once they are out in the open there is tons of potential for us all to get involved and contribute to those that stick.
FYI, another user who contacted us last week for support was referred to your NFS Block and was very happy with the solution. Any feedback we get we will be sure to loop back in to the repository, although as you said yourself it’s working great already.
Lots of progress on the device UI: GitHub - maggie0002/balena-device-ui: A UI for Balena devices.
Still lots planned to make it even easier and simpler to configure and use, including a lighter backend to reduce the size down considerably. As always, if there are ideas, thoughts or bugs would be happy to hear them. I will be updating the readme and Wiki soon, right now it’s a bit behind because of the amount of changes.
3 Likes
This is great @maggie0002 ! Do you think this is ready for use in projects accepting the feature caveats or do you think there is more testing needed by you before we would consider implementing this in our projects?
1 Like
@keenanjohnson, as is so often the case with these things, it all depends on use case. If you are planning on running incubators for babies, then no. Perhaps more realistic criteria relates to how much access you have to your devices. For example, if they are always connected to the internet, then the question of readiness is different because it is easier to send updates as you go, whereas if you want to put things out in a middle of a field, then that may mean you want to wait longer. But that question then also relates to the features you intend to use. If you are just putting the CPU chart on a device, and dropping everything else, then obviously the readiness question changes again.
I realise I am stating the obvious a bit here, let me see if I can be a bit more helpful. Here are some thoughts on things pending that may help making decisions:
- There isn’t any automated testing built in to this yet. This is absent partly because of the way this project evolved, testing wasn’t a high priority when we were experimenting. Now it is absent, because my intent is to use a Golang backend and writing tests now for the Node backend wouldn’t be as valuable. This means that pushing changes will come with more risk, as it involves doing all your tests manually. That said, if you are looking to just show some basic statistics, there probably isn’t much need to push changes, build once and then revisit later.
- A Golang backend will significantly reduce the size of the container, currently at 200mb, I am aiming for closer to 20mb. For many this will have little impact if for example you are preloading, have good internet connections, or intend to have a fairly static UI where you won’t be pushing changes often. I should note though that the plan is not to deprecate the node backend currently used. A new repo will be used to do the Golang backend and frontend on top of it, that way people have the choice of whether they want to work entirely in Typescript (frontend and back) or delve in to mixed languages (Typescript and Golang). The Node backend though isn’t going to evolve much further than where it is in favour of time spent on the new one, so it’s a little more of what you see is what you get, which has both advantages and disadvantages depending on use case (although of course community evolutions always welcome).
- This isn’t a ‘balena supported’ product, this is us working together in the community on things that are useful to users. It means that if you need support or short turnaround on changes or fixes, then it will come down to who is around with the time to take a look.
Perhaps a useful indicator in terms of stages here, is all that is left on my list before adding a tagged version to the hub is to update the docs and to explore an issue where it doesn’t build in the Cloud for older architectures (although I am thinking that may be a cloud builder issue rather than this code, as it builds for all the other architectures ok). There is plenty more that could be done of course, but it is seen as very usable so time to start thinking ahead to the new backend. Any bugs or issues in here, I probably don’t know about them but if you find any then happy to take a look.
All the way back in October it seems like you started the thread. Are you able to edit your initial post and put in a TLDR pointing to the device-ui? 71 posts later, I suspect it will be too much for people to read to get to the project.
It looks like I can’t edit the initial post unfortunately.
Hey guys,
Finally we have the starter interface at a point where it includes all the envisioned initial features. Of course there are areas we would like to tune and new components that we will look to add in the future, but in terms of a base and vision it is now where we were hoping to be. The main new addition, is the ability to configure the interface from a config.yml file, which allows specifying which of the components you would like to be visible and where, and then allows adding it to your existing projects with only a few lines and no build steps. You can check out the documentation Quick Start for more info. The documentation also contains all the info needed to brand it differently, change logos, titles, fonts etc.
Looking forward to hearing any thoughts.
@maggie0002 terrific work!
Would still love to see how we could add authentication, I had done a little digging with Auth0 to see what might be possible without requiring a network connection - for example, if you put the whole UI behind authentication, how do you handle accessing before configuring wifi-connect when you’re still “offline”. But concluded its beyond my technical ability.
Authentication could be extremely useful, especially considering that while playing with local UI, my containers (and secret env variables 
) were publicly accessible!
Very impressed with how far this has come though 
2 Likes
FYI @maggie0002 I just referred someone to this project and they said it was exactly what they were looking for! Great work!
2 Likes