While trying to get renewed certificates picked up by openBalena, I found comments here recommending removing the volumes as part of the container restart process, or discussing how quickstart sets up automatic certificate renewal. This resulted in losing or overwriting the volume that contained the openBalena database, so the old application and its devices are not listed by the Balena CLI. I still have back-ups of the original (still-valid) certificates and activate file with secrets. Is there any way to remotely register the devices with a new application, on the same domain as before, or otherwise remotely connect to them?
Edit: specifically, how to add records for the old devices to the new copy of the database.
I’ve confirmed that devices’ resin_supervisor containers are still attempting to connect and are fine with the still-intact certs:
Thu Apr 28 07:45:52 2022 VERIFY OK: depth=1, CN=vpn-ca.[domain]
Thu Apr 28 07:45:52 2022 VERIFY KU OK
Thu Apr 28 07:45:52 2022 Validating certificate extended key usage
Thu Apr 28 07:45:52 2022 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Thu Apr 28 07:45:52 2022 VERIFY EKU OK
Thu Apr 28 07:45:52 2022 VERIFY OK: depth=0, CN=vpn.[domain]
Thu Apr 28 07:45:52 2022 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 4096 bit RSA
Thu Apr 28 07:45:52 2022 [vpn.[domain]] Peer Connection Initiated with [AF_INET]44.231.88.53:443
Thu Apr 28 07:45:54 2022 SENT CONTROL [vpn.[domain]]: 'PUSH_REQUEST' (status=1)
Thu Apr 28 07:45:54 2022 AUTH: Received control message: AUTH_FAILED
These requests appear to align with open-balena-vpn logs like:
AUTH FAIL: API Authentication failed for c9cd7417dc7ff7c4148e6cb5bde98203
[28/Apr/2022:07:43:30 +0000] "POST /api/v1/auth/ HTTP/1.1" 401 12 "-" "curl/7.64.0"
127.0.0.1:60726 WARNING: Failed running command (--auth-user-pass-verify): external program exited
127.0.0.1:60726 TLS Auth Error: Auth Username/Password verification failed for peer
127.0.0.1:60650 Connection reset, restarting [0]
I am guessing that this is based on one of the keys in /boot/config.json, probably deviceApiKey. If so, what would be the corresponding setting on the openBalena side that I should adjust to let the authentication succeed?
Still curious about where the openvpn credentials are set. In the meantime, attempting to recover volume data from the filesystem. This requires:
Stopping the server instance.
Disconnecting the filesystem from the server instance and mounting it as a secondary disk on another instance (which has enough space to contain any recovered files).
Installing TestDisk on the new server and running testdisk on the attached partition. If there is a warning that “the harddisk seems too small”, continue anyway. Then press p to view the files.
Using the TestDisk interface to navigate to /var/lib/docker/volumes. This shows a mix of recoverable (gray) and damaged (red) directories.
The desired ones are likely mostly overwritten, but it’s possible to select individual ones with : (followed by C), or just press c while a particular volume directory or the whole /var/lib/docker/volumes is highlighted, then choosing a location on the server’s main disk (outside the partition being recovered) to retrieve them and review later.
Any documentation on how the keys in the configuration files fit into the openvpn authentication process would be appreciated.
The server is still using 13GB despite the fresh install, on an operating system that takes under 2GB, so I am hoping to figure out where the extra data is locatedwhich appears to be /var/lib/overlay2 and am wondering whether any of it is recoverable.
src/api.ts in open-balena-vpn receives the POST and converts it into a GET request addressed to the open-balena-api container, using the API’s standard UUID / Bearer Token format.
authDevice() in src/features/vpn/services.ts relays to checkAuth() which builds a query object, again ultimately relying on balena/pinejs. The query appears to go to PineJS’ Auth model, looking for an Auth record with a matching uuid and apiKey. If found, the authentication succeeds.
So, I need to create some auth records in the database. PineJS seems to be configured for PostgreSQL. Can anyone point to the location where those are stored? The PostgreSQL in open-balena-db appears to be empty.psql -U docker -d resin - \dt then lists the expected tables.
Found open-balena-api logging queries based on the key. Not yet clear on the best process for allowing those to succeed.
SELECT "permission"."name"
FROM "permission"
WHERE (EXISTS (
SELECT 1
FROM "api key-has-permission" AS "permission.api key-has-permission"
WHERE "permission"."id" = "permission.api key-has-permission"."permission"
AND EXISTS (
SELECT 1
FROM "api key" AS "permission.api key-has-permission.api key"
WHERE "permission.api key-has-permission"."api key" = "permission.api key-has-permission.ap
AND ("permission.api key-has-permission.api key"."key") IS NOT NULL AND ("permission.api ke
)
)
OR EXISTS (
SELECT 1
FROM "role-has-permission" AS "permission.role-has-permission"
WHERE "permission"."id" = "permission.role-has-permission"."permission"
AND EXISTS (
SELECT 1
FROM "role" AS "permission.role-has-permission.role"
WHERE "permission.role-has-permission"."role" = "permission.role-has-permission.role"."id"
AND EXISTS (
SELECT 1
FROM "api key-has-role" AS "permission.role-has-permission.role.api key-has-role"
WHERE "permission.role-has-permission.role"."id" = "permission.role-has-permission.
AND EXISTS (
SELECT 1
FROM "api key" AS "permission.role-has-permission.role.api key-has-role.api
WHERE "permission.role-has-permission.role.api key-has-role"."api key" = "p
AND ("permission.role-has-permission.role.api key-has-role.api key"."key")
)
)
)
))
ORDER BY "permission"."name" ASC [ '<key>' ]
While I am able to create actor and device records manually, the key/role/permission system is too complex to recreate. Based on the Balena primer, openBalena normally creates all these records during the device provisioning process, and my goal is now to recreate the records produced by the provisioning process without the provisioning key.
Based on the provisionDevice() test function, I found that a device is created with a POST request. For example, with application (fleet) id 1 and device type 56, that comes to:
(or similar - currently getting errors on this request and having to use open-balena-admin). Once the numerical id of the device is known, an API key can be set with a subsequent:
This results in a device record that is linked to an api key record through an actor record, but queries still fail at step 7 when getAPIKey() calls permissions.resolveApiKey(req). Following that to pinejs/src/sbvr-api/permissions.ts leads further to checkApiKey() → getApiKeyPermissions() → $getApiKeyPermissions(). The 10-layer-deep query object there appears to seeks the name of a permission that is granted (via api key-has-permission) to an api key with matching apiKey and null or future expiryDate, or that is granted to a role (via role-has-permission that is granted (via api key-has-role) to an api key that meets those conditions. api key-has-permission is empty, while “role-has-permission” has been painstakingly populated by openBalena, so I focused on choosing a role from the ones that already existed:
So, role id 3 (device-api-key). In fact, the above queries already caused openBalena to create api key-has-role with role=3. After adding a few more devices with the right UUID and key, permissions are being found, and several devices are reported as “Is Online” in the Balena CLI, though only “VPN State: Connected” (while “Connectivity: Unknown”) in open-balena-admin.
The fastest workflow proved to be using open-balena-admin to first add each device with the correct uuid and name, then editing the api key field of the API key record that this automatically created to match the hashes that the open-balena-api server was querying for. This led to everything being marked as connected to the VPN and “online” in Balena CLI, but not “Online” in the admin UI (which is based on the Connectivity state), as above. This connection was sufficient to regain SSH access and allow essential debugging and network inspection tasks. The server also allowed a development build unit to compile and upload a first release.
Restarting the regular group of open-balena containers caused the connections to start being marked as Online in the UI too. A handful of the device records also began showing details like OS version, CPU utilization, temperature, etc. The logs had shown that these were already being reported before, but not accepted by open-balena. The devices whose statuses were now saved tended to also update to the only release, even when the fleet was not set to track this release. I believe that the status reporting might be hindered as long as a device is trying to work through this first update, which is quite demanding on the server since the registry/device cannot find and skip any layers that the device’s (forgotten) previous release and the registry’s (only) new release might have in common. The server load has been quite high due to the messy first upgrades and associated log flood, but over a quarter of the fleet has upgraded so far. Essential capabilities appear to be within reach.
Edit: server load appears to decline proportionally as more devices finish upgrading.
Hi, I lost my complete openbalena setup. En would like to try this solution to build it again. But i can’t find out how i find the correct uid & name for the existing devices. Where can i find these ?
If you lost the list of device UUIDs, you may see them in the open-balena-api container’s log (either regular docker logs or journalctl). The name is arbitrary.
