balena-cli image build process from scratch

neilnz · November 28, 2020, 12:31am

Hi,

I’m trying to build an image configuring/building script to prepare SD card images for our balena devices and seem to have fallen down an undocumented hole. From other posts in this forum, I can see other people have fallen down a similar hole, but I don’t see anyone directly asking how they should do this. I would really expect this to be in the “masterclass” docs, given it’s semi-advanced, but the masterclass makes use of images from the dashboard!

As I originally understood it, I could build a usable image for a BalenaCloud-supported device using these steps:

balena os download --version latest --output balena.img
balena os build-config balena.img raspberrypi4-64 -o config.json
balena os configure balena.img --config config.json --app myapp
balena os preload balena.img --app myapp --commit current

and this would yield a balena.img file that’s roughly the same thing as downloading it from the cloud dashboard, which I could then burn to a card with etcher/dd/etc. Turns out this is false. It yields an image that’ll boot the device into a useless state of not registering with the cloud and not being reachable (because it’s not a dev image). I’m not even sure why balena os configure takes --app as an argument if that’s not the key bit of information it needs to write the app ID into the config in the image…

After a couple of days poking around in the balena source and inside a dev build of the balenaos, I found that balena os initialize (which isn’t really documented) writes additional keys into config.json that tells the OS how to register with the cloud. But it also writes the image to a card at the same time. I don’t want to do that.

I see a few references in this forum to https://github.com/balena-io-modules/balena-register-device which seems to generate this config, in particular the API key. And it appears to be per-device?

I don’t want to burn device-specific IDs into the image because I need to mass-flash these images to cards, preferably not using the balena CLI.

I’d assumed a generic image would generate device-specific UUID and keys on first registration with the cloud (I see some references to this happening in other forum posts). Is this the case? Can I take the same image and burn it to a hundred cards and it’ll register each as a different UUID?

While I can use balena os initialize to burn a card then take an image of that card, everything seems to be discouraging me from doing that, without any clear docs for balena-cli to say how this should happen.

I really think there needs to be “how to build a working balenaos image using the CLI” page in the docs somewhere, that explicitly mentions the role of balena os initialize in making the image usable. I can’t be the only one that’s had this happen to them

Everything else about balena has been smooth and “behaves as expected”, so I really hope I can iron this out and get into prod.

neilnz · November 28, 2020, 1:06am

In the time-honoured fashion of beginning to figure things out just after asking the question, I’ve found another command:

balena config generate

which generates a config.json that includes the necessary keys (applicationId, apiEndpoint, etc) in the JSON, which balena os build-config does not. Combined with the docs at https://github.com/balena-io/balena-cli/blob/master/doc/automated-init.md I’m starting to form an opinion that a config.json from balena os build-config can only be used with either balena os initialize or balena device init, and if you want to use balena os configure with a config.json, it must be generated using balena config generate, not balena os build-config. It seems if you use balena os configure without a config.json, then it will write the app ID etc into the config, but if you pass is a minimal config.json without that stuff, it will just copy that file in verbatim without complaint. Grr.

Update: ref https://github.com/balena-io/balena-cli/blob/8115d156df7e3a957bc451d2f72f1f65ffba8959/lib/commands/os/configure.ts#L239

So, IMHO, there needs to be at minimum an update to balena-cli to warn you when you run balena os configure that your config.json is missing required keys to make a working device and/or will create an unprovisioned device (which I gather is a supported operating mode in some cases). Really it would be more helpful to simply meld in missing application-specific keys for you, like it does when you don’t pass a config.json. Happy to submit a PR if you agree?

pdcastro · November 30, 2020, 1:56pm

@neilnz, many thanks for describing your experience and findings, and sorry for the trouble you went through to figure out a solution. Indeed, the current CLI command set regarding OS image configuration has a few commands that overlap in functionality, and can be confusing. It is a result of contributions over time from developers who viewed the platform from slightly different angles, and had different specific use cases in mind. Some commands like os configure have been refreshed a few times as the platform evolved, while other commands like os build-config appear to be less relevant nowadays. This area of the CLI could indeed do with a review and revamp.

I would really expect this to be in the “masterclass” docs […] but the masterclass makes use of images from the dashboard! […] I really think there needs to be “how to build a working balenaos image using the CLI” page in the docs somewhere

I was looking at section “3. Downloading and Configuring a Provisioning Image using balena CLI” of the Balena CLI Advanced Masterclass and it does seem to cover a set of commands that can configure an image downloaded with balena os download:

$ balena os download fincm3 --version 2.38.0+rev1 --output balena-fin-image.img
$ balena config generate --version 2.38.0+rev1 --application cliApp --appUpdatePollInterval 10 --network wifi --wifiSsid MyNetworkSSID --wifiKey myw1f1n3tw0rk -o wifi-config.json
$ balena os configure --application cliApp --config wifi-config.json balena-fin-image.img

Had you seen that section of the masterclass, and perhaps found that it did not meet your requirements in some way?

I’d assumed a generic image would generate device-specific UUID and keys on first registration with the cloud (I see some references to this happening in other forum posts). Is this the case? Can I take the same image and burn it to a hundred cards and it’ll register each as a different UUID?

I gather that the answer is yes. I have just tested the following set of commands with two Raspberry Pi devices. The same image file was flashed to 2 SD cards. Then I powered both devices and, a few minutes later, both devices showed up in the web dashboard with different UUIDs.

$ balena os download raspberrypi3 --version 2.60.1+rev1.dev --output rpi3-2.60.1+rev1.dev.img
$ balena config generate --version 2.60.1+rev1 --application test-rpi --appUpdatePollInterval 10 --network wifi --wifiSsid MyNetworkSSID --wifiKey myw1f1n3tw0rk -o wifi-config.json
$ balena os configure --application test-rpi --config wifi-config.json rpi3-2.60.1+rev1.dev.img
$ balena util available-drives
$ sudo balena local flash rpi3-2.60.1+rev1.dev.img

I have found that the config.json file produced by balena config generate includes an apiKey attribute, but not a uuid attribute. After the device boots and registers with the cloud, the config.json file in the boot partition is amended to include the uuid and deviceApiKey attributes.

So, IMHO, there needs to be at minimum an update to balena-cli to warn you when you run balena os configure that your config.json is missing required keys to make a working device and/or will create an unprovisioned device (which I gather is a supported operating mode in some cases). Really it would be more helpful to simply meld in missing application-specific keys for you, like it does when you don’t pass a config.json. Happy to submit a PR if you agree?

It sounds good, and your contribution is most welcome! Check the Contributing document (you’ve probably seen it already) for some starting pointers. Compatibility with openBalena also comes to my mind, as I see in the openBalena Getting Started guide that is also uses balena os configure, but I think it would be fine. Thanks again!