Device Type Not Recognized - Fleet Creation Fails on internet isolated Host

openBalena
1

Hi everybody,

I have successfully installed (i hope) open-balena and the balena-cli on the same machine, which is completely isolated from the internet.

When I run the command:

balena fleet create my-fleet-8aio --type revpi-connect-4

return

Device type slug not recognized. Perhaps misspelled?
Check available device types with “balena device-type list”

tl;dr;

Premise:
I installed open-balena successfully in an air-gapped environment. I installed all necessary .deb packages and loaded the open-balena images built on an online machine.

Initial Issue & Resolution:
The stack initially failed because I had forgotten to disable cgroup v2 (which is no longer mentioned in the manual, but was necessary for the version I was using). After disabling cgroup v2 and rebooting the system, all containers started correctly (except for the open-balena-haproxy-sidecar-1 container, which occasionally restarts, but this doesn’t seem to be the primary blocker).

Current Container Status (docker ps output):

lcladm@ansrvgem05:~$ docker ps
CONTAINER ID   IMAGE                                COMMAND                   CREATED       STATUS                       PORTS                                                                                                                       NAMES
317dcea880dc   balena/open-balena-vpn:11.33.8       "/usr/bin/entry.sh"       2 hours ago   Up About an hour (healthy)   80/tcp, 443/tcp, 3128/tcp                                                                                                   open-balena-vpn-1
6b078f4367c0   balena/open-balena-api:43.0.16       "/etc/s6-overlay/scr…"    2 hours ago   Up About an hour (healthy)   80/tcp                                                                                                                      open-balena-api-1
2eb8def76b01   balena/open-balena-registry:2.42.0   "/usr/bin/confd-entr…"    2 hours ago   Up About an hour (healthy)                                                                                                                               open-balena-registry-1
50c66dbe3718   open-balena-cert-manager             "/bin/bash -c /usr/l…"    2 hours ago   Up About an hour             80/tcp, 443/tcp                                                                                                             open-balena-cert-manager-1
81fdacd298dd   open-balena-haproxy-sidecar          "dockerd-entrypoint.…"    2 hours ago   Up About a minute            2375-2376/tcp                                                                                                               open-balena-haproxy-sidecar-1
d240bba19026   balena/ca-private:0.0.15             "/bin/bash -c /usr/l…"    2 hours ago   Up About an hour (healthy)   8888/tcp                                                                                                                    open-balena-balena-ca-1
12a902bcfd61   balena/open-balena-db:6.0.0          "/balena-entrypoint.…"    2 hours ago   Up About an hour (healthy)   5432/tcp                                                                                                                    open-balena-db-1
f8aa155831f6   docker:dind                          "/bin/sh -c 'set -x\n…"   2 hours ago   Up About an hour (healthy)   2375-2376/tcp                                                                                                               open-balena-docker-1
e25607280bb0   open-balena-haproxy                  "docker-entrypoint.s…"    2 hours ago   Up About an hour (healthy)   0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp, 0.0.0.0:1936->1936/tcp, [::]:1936->1936/tcp   open-balena-haproxy-1
aab4db854779   redis:8.2-alpine                     "docker-entrypoint.s…"    2 hours ago   Up About an hour (healthy)   6379/tcp                                                                                                                    open-balena-redis-1
94686cf938f3   balena/open-balena-s3:2.28.92        "/usr/bin/entry.sh"       2 hours ago   Up About an hour (healthy)   80/tcp                                                                                                                      open-balena-s3-1

The command:

balena device-type list

returns an empty list, but even on the online machine I used to create the balena images, the same command returns an empty list, yet I can create fleets correctly there. I don’t understand what is going wrong here.

What checks can I perform?
Thank you.
Andrea

2

Using --debug option the result is the following

lcladm@ansrvgem05:~$ balena fleet create gems-c4-4aio --type revpi-connect-4 --debug
[debug] new argv=[/home/lcladm/balena-cli/bin/node,/home/lcladm/balena-cli/bin/run,fleet:create,gems-c4-4aio,--type,revpi-connect-4] length=6
[debug] Deprecation check: 0.74840 days since last npm registry query for next major version release date.
[debug] Will not query the registry again until at least 7 days have passed.
[debug] Event tracking error: getaddrinfo ENOTFOUND data.MY_DOMAIN.local
Device type slug not recognized. Perhaps misspelled?
Check available device types with "balena device-type list"

BalenaInvalidDeviceType: Invalid device type: revpi-connect-4
    at Object.get (/home/lcladm/balena-cli/node_modules/balena-sdk/es2018/models/device-type.js:158:23)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async /home/lcladm/balena-cli/node_modules/balena-sdk/es2018/models/application.js:485:28
    at async Promise.all (index 0)
    at async Object.create (/home/lcladm/balena-cli/node_modules/balena-sdk/es2018/models/application.js:506:52)
    at async Object.applicationCreateBase (/home/lcladm/balena-cli/build/utils/application-create.js:12:29)
    at async FleetCreateCmd.run (/home/lcladm/balena-cli/build/commands/fleet/create.js:8:9)
    at async FleetCreateCmd._run (/home/lcladm/balena-cli/node_modules/@oclif/core/lib/command.js:181:22)
    at async Config.runCommand (/home/lcladm/balena-cli/node_modules/@oclif/core/lib/config/config.js:456:25)
    at async run (/home/lcladm/balena-cli/node_modules/@oclif/core/lib/main.js:97:16)
3

Hi everyone,

I managed to solve the issue.
It turns out my offline database was empty.
I fixed it by dumping the following tables from an online machine (where fleet creation was working) and importing them into the offline one:

On the online machine:

docker exec open-balena-db-1 pg_dump -U docker -d resin -t '"cpu architecture"' --data-only --inserts > 01_cpu_arch.sql
docker exec open-balena-db-1 pg_dump -U docker -d resin -t '"device manufacturer"' --data-only --inserts > 02_manufacturer.sql
docker exec open-balena-db-1 pg_dump -U docker -d resin -t '"device family"' --data-only --inserts > 03_family.sql
docker exec open-balena-db-1 pg_dump -U docker -d resin -t '"device type"' --data-only --inserts > 04_device_types.sql
docker exec open-balena-db-1 pg_dump -U docker -d resin -t '"device type alias"' --data-only --inserts > 05_aliases.sql

On the offline machine:

cat 01_cpu_arch.sql | docker exec -i open-balena-db-1 psql -U docker -d resin
cat 02_manufacturer.sql | docker exec -i open-balena-db-1 psql -U docker -d resin
cat 03_family.sql | docker exec -i open-balena-db-1 psql -U docker -d resin
cat 04_device_types.sql | docker exec -i open-balena-db-1 psql -U docker -d resin
cat 05_aliases.sql | docker exec -i open-balena-db-1 psql -U docker -d resin

After importing the device type alias table (which was the missing link for the CLI/SDK), I was finally able to create the fleet successfully.

I would like to understand when these tables are supposed to be populated during a standard setup. In my case, they were completely empty. I wonder if this is related to an initial boot issue I had with incorrect cgroup settings that prevented containers from starting, or if something else went wrong during the setup phase.

Any insight or suggested checks to verify the root cause would be greatly appreciated.

Thanks to everyone for the support!
Andrea

4

I would like to understand when these tables are supposed to be populated during a standard setup.

Hello, these tables are populated by the initialisation of open-balena-api on startup/init from public AWS/S3. If the API doesn’t have Internet connectivity on startup, it will never initialise this list. What you’ve done, by manually importing DB content, IMO is a valid workaround.

openBalena was not designed to be air-gapped.

Cheers!

5

Hi ab77,

I see.
In the end, I wasn’t able to complete the offline installation because once that error was resolved, the CLI started throwing timeout errors during fleet creation (specifically, balena fleet create was timing out). I checked the logs for all containers and mapped out the request chain (aided by the --debug flag and source code investigation), then simulated the calls between containers using curl. The result was always a timeout. I couldn’t find a way to resolve it.

Therefore, I proceeded to perform the installation on an internet-connected Linux VM, generating the fleets and JSON configurations while still online, and then moved the VM into the isolated environment.

[Off-topic] Currently, I’m having issues setting up a new device, so I can’t verify yet if it will connect correctly to open-balena and if open-balena will work in the isolated environment.

Even though it isn’t designed to work in an air-gapped environment, I hope it’s still usable.
Or do you happen to know of any specific blockers?

Thanks, Andrea

6

Have you looked at this (stale) PR: DRAFT: Enable Air-gapped open-balena deployment by compiaffe · Pull Request #241 · balena-io/open-balena · GitHub ?

It may provide some inspiration…

Cheers!