Using balenaOS securely (production Image) offline

[nmaas87]

Ok, as I have a lot of confusion right now I need to ask to get this resolved
My usecase: Use the production type image without balenaCloud or other things.

So I download the latest production image from balenaOS - Run Docker containers on embedded IoT devices, then set the config.json before the first boot:

{
    "deviceType": "raspberrypi4-64",
    "persistentLogging": false,
    "hostname": "rpi01",
    "developmentMode": false,
    "country": "DE",
    "ntpServers": "ptbtime1.ptb.de ptbtime2.ptb.de ptbtime3.ptb.de",
    "dnsServers": "1.1.1.1 1.0.0.1",  
    "os": {
      "sshKeys": [
        "myPublicSSHKey"
      ]
    }
  }

after booting, I make a balena scan

  host:      rpi01.local
  address:   192.168.4.200
  osVariant: production

so far, so good. Trying to access balenaOS does not work

balena ssh rpi01.local USERNAME@rpi01.local: Permission denied (publickey).
SSH: Process exited with non-zero status code "255"

Are the SSH keys correctly configured in balenaCloud? See:
https://www.balena.io/docs/learn/manage/ssh-access/#add-an-ssh-key-to-balenacloud
Ar`?

obviously, it cannot connect, because balenaCLI does not know the private key I used. Accessing it manually does work

ssh 192.168.4.200 -lroot -p22222 -i ..\rpi01priv

So, now the big question: How can I add my local ssh private key into balenaCLI for use, so that I can locally build and push to this device without a) using balenaCloud or other external services (yeah, I need internet for downloading from Docker Hub etc, that is understood ) and without using b) a development image (does not matter if configured in local mode or not, development images are evil )

Any thoughts?
Thanks!

[nmaas87]

Obviously, the most important thing would be the balena push with this device

[nmaas87]

Any ideas? Would need to deploy some devices in a day

[mpous]

@nmaas87 did you try a balena push <local ip address>?

balena ssh goes through balenaCloud and balena push <ip> goes directly to the device.

Let us know if that works!

[nmaas87]

Thanks Marc, yes I tried and it did not work - and as a matter of fact it makes sense that it cannot work like this:
a) Its a production image and I set a custom ssh key (which balena cli does not know anything about)
b) So balena cli can reach the RPi, but cannot login (see error above, same error happens if used with the real ip)
c) Hence it does not know how and which private key to use, it cannot push to the device.

So I guess the biggest problem would be to introduce the private key on my locale (Windows 10 x.x) machine to balenaCLI so that it can do its job… Any ideas?

Thanks for the help :)!

[mpous]

@nmaas87 could you please tell us more about your use case? why do you want to do local push on production OS images?

[nmaas87]

I need to run completely air-gapped balenaOS devices in a production environment. During some setup operation I can deal with having it only access the Docker Hub, but not more. Due to the nature of the source code involved we got an absolute ban on uploading it to any resource, so even a “just create a fleet, upload and disconnect the device / use it in the airgapped environment” is out of the question - so only a “real” local push with all VPN instances firewalled / blocked is ok. openBalena cannot be used due to airgapped nature and workload. Production OS needs to be used as development images are wide open and will not survive any security audit / will be removed asap. I tried in the past to harden balena development devices, but actually it does not make any sense - because the correct configuration is there and called “production”

[mpous]

@nmaas87 the only supported mechanism for offline updates is described in Offline updates - Balena Documentation. This involves downloading and preloading an OS image, and reflashing the device on site.

Unfortunately application-only offline updates are not currently supported.

I understand that you don’t have access to the SD card, to change the config.json right?

[nmaas87]

Thank you Marc, no the Offline Update mechanism is also out of realm of possibilities, as it still would need me to create a fleet online and hence sending the sourcecode to the cloud, so a big no no.

I have total access to the SD card as long as I did not implemented the device - so you can shot me any idea

I tried something crazy and I am a bit stunned that it did work: I flashed the latest rpi4 production image and generate the custom config.json before the first boot as mentioned above.

{
    "deviceType": "raspberrypi4-64",
    "persistentLogging": false,
    "hostname": "rpi01",
    "developmentMode": false,
    "country": "DE",
    "ntpServers": "ptbtime1.ptb.de ptbtime2.ptb.de ptbtime3.ptb.de",
    "dnsServers": "1.1.1.1 1.0.0.1",  
    "os": {
      "sshKeys": [
        "myPublicSSHKey"
      ]
    }
  }

I just saw it cannot connect to 2375, which is the balenaEngine port - which makes total sense, as in production mode, this is blocked…

I logged manually into the device with a quick
ssh 192.168.4.200 -lroot -p2222 -i sshkey
then I changed the /mnt/boot/config.json and set developmentMode true and did a quick reboot now. After the device came back online I could push an image… I really though the supervisor would be able to tell that I changed the config.json, restore the corruption from its sqlite database and reboot - but it did not - so I pushed my image… changed developmentMode to false again, rebooted - and bam - image still there and running. However, I guess this is more of an error and should not work - even if its damn nice that it does

[mpous]

Actually this is what a developer from balenaOS told me to tell you to experiment, but i completely forgot

test more and let us know if that works for you!

[mpous]

oh no @nmaas87 you are moving your OS image into development doing this! But maybe it’s ok to have development during some minuts and move back to Production?

could you please try developmentMode: false and localMode: true?

[nmaas87]

Yeah I know, it is not perfect but better than nothing.

dev false and local true did not do anything, did not work out → Same error.

I had one very silly idea:

 ssh 192.168.4.200 -vvv -lroot -p22222 -i .ssh/sshkey -L 2375:10.114.104.1:2375 -L 48484:10.114.104.1:48484 -L 22222:127.0.0.1:22222

Afterwards:
balena push 127.0.0.1

The good thing: Yeah, you can talk to the 48484 daemon like that. But the balenaEngine does not expose any port in production mode, not even internally. So even redirecting ports into the machine will not work:

root@rpi01:~# netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.2:53            0.0.0.0:*               LISTEN
tcp        0      0 10.114.102.1:53         0.0.0.0:*               LISTEN
tcp        0      0 :::22222                :::*                    LISTEN
tcp        0      0 :::48484                :::*                    LISTEN
udp        0      0 0.0.0.0:1234            0.0.0.0:*
udp        0      0 0.0.0.0:5353            0.0.0.0:*
udp        0      0 0.0.0.0:64967           0.0.0.0:*
udp        0      0 127.0.0.2:53            0.0.0.0:*
udp        0      0 10.114.102.1:53         0.0.0.0:*
udp        0      0 127.0.0.1:323           0.0.0.0:*
udp        0      0 :::1234                 :::*
udp        0      0 :::5353                 :::*
udp        0      0 :::49255                :::*
udp        0      0 ::1:323                 :::*
udp        0      0 fe80::cc3:b959:8f6b:fffd:546 :::*  

So at the moment looks like I have to go with switching between both modes hm… But if you got some ideas, just keep em coming, going to check back tomorrow

[nmaas87]

Ok, another issue: How to set HOST_CONFIG settings?

I added multiple parameters which should be picked up by the supervisor and included in the config.txt - but nothing happens with them?

Any idea?

version: '2'
services:
  specialsensor:
    #image: balenablocks/sensor
    build: .
    privileged: true
    environment:
      - MQTT_ADDRESS=x
      - MQTT_PUB_INTERVAL=10
      - MQTT_PUB_TOPIC=x
      - MQTT_USE_TLS=1
      - MQTT_PORT=8883
      - MQTT_USERNAME=x
      - MQTT_PASSWORD=1
      - COLLAPSE_FIELDS=1
      - RAW_VALUES=0
      - BALENA_HOST_CONFIG_dtoverlay="disable-bt"
      - RESIN_HOST_CONFIG_disable_splash=1
      - RESIN_HOST_CONFIG_gpu_mem=192
      - BALENA_HOST_CONFIG_hdmi_group=2
      - BALENA_HOST_CONFIG_hdmi_mode=85
      - BALENA_HOST_CONFIG_display_hdmi_rotate=1
      - BALENA_HOST_CONFIG_hdmi_force_hotplug=1
      - BALENA_HOST_CONFIG_enable_uart=1
    labels:
      io.balena.features.kernel-modules: '1'
      io.balena.features.sysfs: '1'
      io.balena.features.supervisor-api: '1'
    ````

[nmaas87]

Well, I did solve it in that way that I set the config.txt to the correct values before first boot, so that I did not have to touch the sqlite database on the balenaOS.

I have now deployed two RPi 4 and have one additional RPi 4 and one RPi 3 (aarch64) already prepared and going to field-deploy tomorrow. Thats a good thing and I guess more will follow as soon as we got this RPi-Crisis over with. But the results from the field are already very impressive and I think the sensors I deployed show some irregularities we are currently already stretching our heads about, even with only two days of recorded data.

[mpous]

How is this running @nmaas87 ? sorry i didn’t get your pings here?

[nmaas87]

Hey Marc, this did work out great so far. I got 3x RPi 4-64 and 1x RPi 3-64 with mixed workloads (all working with BME280, some with SGP30 for adding additional use to making lab spaces more secure and tech spaces more supervised - and running flight maps of the ISS or mission clocks in our controlroom). As soon as I get more RPis (currently backordered) this is going to increase. But yeah, so far so good. I am going to have to think about an update path / allow for ease of updating the balenaOS in an more offline fashion. Other than that, I am good. Loads better than the old design I threw in there a year ago with systems in developer mode. This was a horrible idea back then…

[mpous]

@nmaas87 did you try the offline updates?

I’m looking forward to get your point of view of the offline updates

[nmaas87]

@mpous
I am not really sure if my understanding is correctly (or if the options have changed, because the website looks a bit different than the last time I visited :D) - but in the back of my mind I had the little voice that you’d need to push your code to balenaCloud first so that you could create an offline update / preload from it. Is that still the case? Because we cannot push this code to any online service, only locally work on e.g. our own laptop is allowed.

[mpous]

Indeed! instructions are here Offline updates - Balena Documentation

Looking forward to get your feedback when you test this

[nmaas87]

I did study the instructions but it is sadly still not viable for our usecase as it is necessary to push code to balenaCloud / deploy to balenaCloud for the offline update to work. With that I cannot use this technique but would resort to recreating SD like on initial deployment and swapping them out.

Sorry!