open-balena-admin: an admin interface for openbalena

drcnyc · June 24, 2022, 1:36am

@the-real-kenna thank you for all the information - it is very helpful as we continue to develop open-balena-admin. Can you (or your engineers) provide some additional details on the following:

Pine assumes it has direct and exclusive control of the database so in theory it can change the layout of the database in completely arbitrary ways at any time.

Does this mean that it can actually change the table structure while running? Or are you referring to changes occurring from one release to the next? If the former, that’s an issue. If the latter, we have a version management system within open-balena-admin which based on the version of open-balena-api that is running (specified during the install process) it will modify the underlying table structure.

the release.semver (which is also a computed term) + release.revision pair

the image location (which is the basis for our registry to work & devices to download updates)

Are these items referring to deployment of apps / creation of new images and releases? If so, we only interface with images / releases on a read only basis.

Other than the above items, it sounds like it’s essentially pre-commit “soft” validation (as “hard” rules are enforced by the database, i.e. text field length limits, numbers vs text, foreign keys, etc.), computed fields and permissions that would need to be replicated if bypassing pine. We have some basic soft validation implemented in the front end now, but not much in terms of computed fields. Is there a full list of computed fields available anywhere?

Separately, I am happy to explore ways to integrate Pine (or any other balena best practices for that matter) into open-balena-admin - I don’t suppose this is something that balena would be willing to help with? I believe there is a community of users who are either using openbalena in production (and muddling through the limitations) or looking to do so but not able to get over the limitations - and I believe a tool such as this could bridge the gap. Either way, we will continue to develop it, and provide the latest updates to the community in the hopes that at least some members are finding it helpful despite its flaws.

the-real-kenna · June 24, 2022, 2:01am

@drcnyc I don’t know the answers, but will ping the folks who replied to me and ask if they can elaborate more here in-thread while I’m out of office the next few days.

Hopefully we can help get you pointed in the right direction for these specific questions, but also come up with some good ways to share information better between our strategies / intents with openBalena and the community that uses and builds on top of it.

the-real-kenna · June 24, 2022, 2:16am

I also meant to say, the reason I’m sharing this information in part is also because we are really excited to see what you’ve created on openBalena and want to be sure it continues to be successful. I hope that comes across in these async messages, but wanted to be explicit about it just in case. I’ve personally recommended your UI to several customers already, so please know we’re excited about what you’ve built and are hoping to support you in that work remaining successful over time.

thgreasi · June 24, 2022, 11:12am

Hi @drcnyc

Does this mean that it can actually change the table structure while running?

No. At the moment we only change the database schema in migrations that run when a new version of open-balena-api first starts up.

Are these items referring to deployment of apps / creation of new images and releases?

~Yes. The release.revision & image.is_stored_at__image_location fields are populated by the open-balena-api when creating a new release or image. I you are just surfacing the DB data for those columns in a read-only way then there shouldn’t be an issue. That being said that, there are the release.version & release.raw_version which are computed term that the balenaCloud dashboard surfaces in the releases page, which you would need to re-implement.

permissions that would need to be replicated if bypassing pine

Keep in mind that Pine also has permission caching & invalidation, which might be an issue if you start modifying the DB directly.

Is there a full list of computed fields available anywhere?

The lazily computed (on request) terms are:

release.semver, release.version, release.is_final & release.raw_version
device.overall_status & device.overall_progress
the user__has_direct_access_to__application computed resource/table

https://github.com/balena-io/open-balena-api/search?q="const+addToModel"

On top of these, there are also “eager”/on-write computed terms, such as:

the release.revision
the invalidation of the user’s jwt when the username/password gets updated
generating a slug for each application created or renamed
creating a unique image location for each new image

pre-commit “soft” validation (as “hard” rules are enforced by the database

There is also a group of rules which are not enforced by the database, but I’m hesitant to categorize them as soft, since violating them would result the DB to be in an invalid state. That group includes rules like like making sure devices only run releases that are of status “success”, which pine confirms with the database before it persists any uncommitted data that might invalidate those rules.
Here is a list of such rules which atm are enforced by Pine and not the DB:

github.com

balena-io/open-balena-api/blob/a7e5480826b4934c159ffc5e500555ecfce8d3a2/src/balena.sbvr#L850-L865


      
          Rule: It is necessary that each device1 that is managed by a device2, belongs to an application1 that depends on an application2 that owns the device2.
          Rule: It is necessary that each release that has a release version1 and has a status that is equal to "success" and is not invalidated, belongs to an application that owns exactly one release that has a release version2 that is equal to the release version1 and has a status that is equal to "success" and is not invalidated.
          Rule: It is necessary that each image that has a status that is equal to "success", has a push timestamp.
          Rule: It is necessary that each application that is public, has an application type that is not legacy.
          Rule: It is necessary that each application that owns a release1 that has a status that is equal to "success" and has a commit1, owns at most one release2 that has a status that is equal to "success" and has a commit2 that is equal to the commit1.
          Rule: It is necessary that each application that owns a release1 that has a revision, owns at most one release2 that has a semver major that is of the release1 and has a semver minor that is of the release1 and has a semver patch that is of the release1 and has a semver prerelease that is of the release1 and has a variant that is of the release1 and has a revision that is of the release1.
          Rule: It is necessary that each release that should be running on a device, has a status that is equal to "success" and belongs to an application1 that the device belongs to.
          Rule: It is necessary that each release that should be running on an application, has a status that is equal to "success" and belongs to the application.
          Rule: It is necessary that each application type that does not supports multicontainer, is of no application that owns a release that contains at least 2 images.
          Rule: It is necessary that each release that should operate a device, has a status that is equal to "success".
          Rule: It is necessary that each release that should operate a device that is of a device type, belongs to an application that is host and is for the device type.
          -- native supervisor release rules, separated for legibility
          Rule: It is necessary that each release that should manage a device, has a status that is equal to "success" and has a semver major that is greater than 0 or has a semver minor that is greater than 0 or has a semver patch that is greater than 0.
          -- this rule is meant to prevent accidentally setting a host extension as a supervisor (i.e., another public + non-host app)
          Rule: It is necessary that each release that should manage a device, belongs to an application that is public and is not host and has a slug that is equal to "balena_os/aarch64-supervisor" or "balena_os/amd64-supervisor" or "balena_os/armv7hf-supervisor" or "balena_os/i386-supervisor" or "balena_os/i386-nlp-supervisor" or "balena_os/rpi-supervisor".
          Rule: It is necessary that each release that should manage a device that is of a device type1, belongs to an application that is for a device type2 that is of a cpu architecture that is supported by the device type1.

A good way to get introduced to Pine or at least its exposed OData endpoints, would be through the balena-sdk, which is solely what our dashboard uses to interact with our backend.

Kind regards,
Thodoris

drcnyc · June 24, 2022, 5:26pm

@thgreasi thank you!!! This is extremely helpful information and will drive the next leg of development of open-balena-admin. I’m still taking a lot of this in but below are my thoughts / action items.

No. At the moment we only change the database schema in migrations that run when a new version of open-balena-api first starts up.

Great - so we should be good on the table / column structure, it just means that we need to keep an eye on any changes to the sbvr with each new release of open-balena-api, which is exactly how we built our backwards compatibility with multiple versions. Whenever the sbvr changes, we simply need to work that change into our versioning system so we can support the new feature. But it wouldn’t break anything on running instances, this would only apply to someone who intentionally updates their open-balena-api instance, so it would just necessitate a concurrent update of open-balena-admin.

~Yes. The release.revision & image.is_stored_at__image_location fields are populated by the open-balena-api when creating a new release or image. I you are just surfacing the DB data for those columns in a read-only way then there shouldn’t be an issue. That being said that, there are the release.version & release.raw_version which are computed term that the balenaCloud dashboard surfaces in the releases page, which you would need to re-implement.

Also great - we intentionally are not involved with images and releases. Similar to balena cloud, open-balena-admin users need to use balena-cli for release deployment.

The lazily computed (on request) terms are:

release.semver, release.version, release.is_final & release.raw_version

device.overall_status & device.overall_progress

the user__has_direct_access_to__application computed resource/table

ACTION ITEM: These computed fields should be implemented to show on their respective pages.

On top of these, there are also “eager”/on-write computed terms, such as:

the release.revision

the invalidation of the user’s jwt when the username/password gets updated

generating a slug for each application created or renamed

creating a unique image location for each new image

Release revision is not necessary because we don’t support pushing releases via the web. We have our own invalidation process for the jwt which actually uses open-balena-api so should follow this already. For the slug, we allow users to create their own slug and make it required. And again we don’t need to worry about the image location because we don’t support pushing images via the web.

There is also a group of rules which are not enforced by the database, but I’m hesitant to categorize them as soft, since violating them would result the DB to be in an invalid state. That group includes rules like like making sure devices only run releases that are of status “success”, which pine confirms with the database before it persists any uncommitted data that might invalidate those rules.
Here is a list of such rules which atm are enforced by Pine and not the DB:

I did see those when combing through the sbvr but did not appreciate that they were not implemented at the database level - but when reading through them it makes total sense. Thank you for clarifying this!! It basically gives me a to-do list to work through implementing the same logic in our front end. Some of them seem pretty obscure and I would think unlikely for most users to encounter (i.e. if device1 is managed by device2 then application1 needs to depend on application2, public application types can not be legacy, etc. But others are more relevant like you must have a successful release tied to a device.

ACTION ITEM: These rules need to be implemented in open-balena-admin, and monitored for changes in the same way as the database structure.

This is the single most helpful contribution I have received for this project - and I can’t thank you enough for it.

matialvarezs · March 12, 2024, 6:20pm

@nmaas87 @drcnyc @the-real-kenna

Hello,

I’ve encountered an issue when starting services with Docker Compose for the open-balena-admin interface. After initiating the services and navigating to the IP and port 8080 in my browser, I’m presented with a JavaScript error in the console:

Uncaught TypeError: Cannot read properties of undefined (reading ‘resources’)

This error prevents the interface from loading properly. I followed the standard Docker Compose setup instructions without making modifications to the codebase. Could you please provide guidance on how to resolve this issue?

Thank you.
Regards
Matias

drcnyc · March 30, 2024, 6:18pm

@matialvarezs just responded to your GitHub issue to this effect - hopefully someone who is using docker compose for deployment will be able to help. Also note that we recently pushed a major update to open-balena-ui thanks to a significant contribution from a community member so it’s probably worth giving that a shot once we build new images - which will be happening shortly…

And while I know there is a learning curve associated with kubernetes, I would recommend giving that a shot using the helm scripts available here - it’s fairly easy to enable kubernetes in your local docker environment, and deployment is much easier in my opinion, once you get your head around how kubernetes works.