# Using balenaOS securely (production Image) offline

**URL:** <https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526>\
**Category:** Product support\
**Created:** [March 14, 2022, 4:26pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526 "2022-03-14T16:26:13Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![nmaas87](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/nmaas87/32/5570_2.png) [@nmaas87](https://forums.balena.io/u/nmaas87)\
**Post date:** [March 14, 2022, 4:26pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/1 "2022-03-14T16:26:13Z")

</div>

Ok, as I have a lot of confusion right now I need to ask to get this resolved 😃  
My usecase: Use the production type image without balenaCloud or other things.

So I download the latest production image from [balenaOS - Run Docker containers on embedded IoT devices](http://balena.io/os), then set the config.json before the first boot:

```auto
{
    "deviceType": "raspberrypi4-64",
    "persistentLogging": false,
    "hostname": "rpi01",
    "developmentMode": false,
    "country": "DE",
    "ntpServers": "ptbtime1.ptb.de ptbtime2.ptb.de ptbtime3.ptb.de",
    "dnsServers": "1.1.1.1 1.0.0.1",  
    "os": {
      "sshKeys": [
        "myPublicSSHKey"
      ]
    }
  }

```

after booting, I make a balena scan

```auto
  host: rpi01.local
  address: 192.168.4.200
  osVariant: production

```

so far, so good. Trying to access balenaOS does not work

```auto
balena ssh rpi01.local USERNAME@rpi01.local: Permission denied (publickey).
SSH: Process exited with non-zero status code "255"

Are the SSH keys correctly configured in balenaCloud? See:
https://www.balena.io/docs/learn/manage/ssh-access/#add-an-ssh-key-to-balenacloud
Ar`?

```

obviously, it cannot connect, because balenaCLI does not know the private key I used. Accessing it manually does work

```auto
ssh 192.168.4.200 -lroot -p22222 -i ..\rpi01priv

```

So, now the big question: How can I add my local ssh private key into balenaCLI for use, so that I can locally build and push to this device without a) using balenaCloud or other external services (yeah, I need internet for downloading from Docker Hub etc, that is understood 🙂 ) and without using b) a development image (does not matter if configured in local mode or not, development images are evil 😉 )

Any thoughts? 🙂  
Thanks!

---

<div class="post-metadata">

**Author:** ![nmaas87](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/nmaas87/32/5570_2.png) [@nmaas87](https://forums.balena.io/u/nmaas87)\
**Post date:** [March 14, 2022, 4:26pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/2 "2022-03-14T16:26:46Z")

</div>

Obviously, the most important thing would be the `balena push` with this device 😉

---

<div class="post-metadata">

**Author:** ![nmaas87](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/nmaas87/32/5570_2.png) [@nmaas87](https://forums.balena.io/u/nmaas87)\
**Post date:** [March 15, 2022, 10:27am UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/3 "2022-03-15T10:27:21Z")

</div>

Any ideas? Would need to deploy some devices in a day 🙂

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [March 15, 2022, 12:24pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/4 "2022-03-15T12:24:55Z")

</div>

@nmaas87 did you try a `balena push <local ip address>`?

`balena ssh` goes through balenaCloud and `balena push <ip>` goes directly to the device.

Let us know if that works!

---

<div class="post-metadata">

**Author:** ![nmaas87](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/nmaas87/32/5570_2.png) [@nmaas87](https://forums.balena.io/u/nmaas87)\
**Post date:** [March 15, 2022, 12:43pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/6 "2022-03-15T12:43:58Z")

</div>

Thanks Marc, yes I tried and it did not work - and as a matter of fact it makes sense that it cannot work like this:  
a) Its a production image and I set a custom ssh key (which balena cli does not know anything about)  
b) So balena cli can reach the RPi, but cannot login (see error above, same error happens if used with the real ip)  
c) Hence it does not know how and which private key to use, it cannot push to the device.

So I guess the biggest problem would be to introduce the private key on my locale (Windows 10 x.x) machine to balenaCLI so that it can do its job… Any ideas?

Thanks for the help :)!

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [March 15, 2022, 12:50pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/7 "2022-03-15T12:50:20Z")

</div>

@nmaas87 could you please tell us more about your use case? why do you want to do `local push` on production OS images?

---

<div class="post-metadata">

**Author:** ![nmaas87](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/nmaas87/32/5570_2.png) [@nmaas87](https://forums.balena.io/u/nmaas87)\
**Post date:** [March 15, 2022, 1:24pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/9 "2022-03-15T13:24:43Z")

</div>

I need to run completely air-gapped balenaOS devices in a production environment. During some setup operation I can deal with having it only access the Docker Hub, but not more. Due to the nature of the source code involved we got an absolute ban on uploading it to any resource, so even a “just create a fleet, upload and disconnect the device / use it in the airgapped environment” is out of the question - so only a “real” local push with all VPN instances firewalled / blocked is ok. openBalena cannot be used due to airgapped nature and workload. Production OS needs to be used as development images are wide open and will not survive any security audit / will be removed asap. I tried in the past to harden balena development devices, but actually it does not make any sense - because the correct configuration is there and called “production” 🙂

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [March 15, 2022, 2:34pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/10 "2022-03-15T14:34:25Z")

</div>

@nmaas87 the only supported mechanism for offline updates is described in [Offline updates - Balena Documentation](https://www.balena.io/docs/learn/deploy/offline-updates/). This involves downloading and preloading an OS image, and reflashing the device on site.

Unfortunately application-only offline updates are not currently supported.

I understand that you don’t have access to the SD card, to change the `config.json` right?

---

<div class="post-metadata">

**Author:** ![nmaas87](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/nmaas87/32/5570_2.png) [@nmaas87](https://forums.balena.io/u/nmaas87)\
**Post date:** [March 15, 2022, 2:44pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/11 "2022-03-15T14:44:51Z")

</div>

Thank you Marc, no the Offline Update mechanism is also out of realm of possibilities, as it still would need me to create a fleet online and hence sending the sourcecode to the cloud, so a big no no.

I have total access to the SD card as long as I did not implemented the device 🙂 - so you can shot me any idea 🙂

I tried something crazy and I am a bit stunned that it did work: I flashed the latest rpi4 production image and generate the custom config.json before the first boot as mentioned above.

```auto
{
    "deviceType": "raspberrypi4-64",
    "persistentLogging": false,
    "hostname": "rpi01",
    "developmentMode": false,
    "country": "DE",
    "ntpServers": "ptbtime1.ptb.de ptbtime2.ptb.de ptbtime3.ptb.de",
    "dnsServers": "1.1.1.1 1.0.0.1",  
    "os": {
      "sshKeys": [
        "myPublicSSHKey"
      ]
    }
  }

```

I just saw it cannot connect to 2375, which is the balenaEngine port - which makes total sense, as in production mode, this is blocked…

I logged manually into the device with a quick  
`ssh 192.168.4.200 -lroot -p2222 -i sshkey`  
then I changed the `/mnt/boot/config.json` and set developmentMode true and did a quick `reboot now`. After the device came back online I could push an image… I really though the supervisor would be able to tell that I changed the config.json, restore the corruption from its sqlite database and reboot - but it did not 😃 - so I pushed my image… changed developmentMode to false again, rebooted - and bam - image still there and running. However, I guess this is more of an error and should not work - even if its damn nice that it does 😃

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [March 15, 2022, 2:58pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/12 "2022-03-15T14:58:48Z")

</div>

Actually this is what a developer from balenaOS told me to tell you to experiment, but i completely forgot 🤦‍♂️

test more and let us know if that works for you!

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [March 15, 2022, 3:01pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/13 "2022-03-15T15:01:30Z")

</div>

oh no @nmaas87 you are moving your OS image into development doing this! But maybe it’s ok to have development during some minuts and move back to Production?

could you please try `developmentMode: false` and `localMode: true`?

---

<div class="post-metadata">

**Author:** ![nmaas87](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/nmaas87/32/5570_2.png) [@nmaas87](https://forums.balena.io/u/nmaas87)\
**Post date:** [March 15, 2022, 3:31pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/14 "2022-03-15T15:31:43Z")

</div>

Yeah I know, it is not perfect but better than nothing.

dev false and local true did not do anything, did not work out → Same error.

I had one very silly idea:

```auto
 ssh 192.168.4.200 -vvv -lroot -p22222 -i .ssh/sshkey -L 2375:10.114.104.1:2375 -L 48484:10.114.104.1:48484 -L 22222:127.0.0.1:22222

```

Afterwards:  
`balena push 127.0.0.1`

The good thing: Yeah, you can talk to the 48484 daemon like that. But the balenaEngine does not expose any port in production mode, not even internally. So even redirecting ports into the machine will not work:

```auto
root@rpi01:~# netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 127.0.0.2:53 0.0.0.0:* LISTEN
tcp 0 0 10.114.102.1:53 0.0.0.0:* LISTEN
tcp 0 0 :::22222 :::* LISTEN
tcp 0 0 :::48484 :::* LISTEN
udp 0 0 0.0.0.0:1234 0.0.0.0:*
udp 0 0 0.0.0.0:5353 0.0.0.0:*
udp 0 0 0.0.0.0:64967 0.0.0.0:*
udp 0 0 127.0.0.2:53 0.0.0.0:*
udp 0 0 10.114.102.1:53 0.0.0.0:*
udp 0 0 127.0.0.1:323 0.0.0.0:*
udp 0 0 :::1234 :::*
udp 0 0 :::5353 :::*
udp 0 0 :::49255 :::*
udp 0 0 ::1:323 :::*
udp 0 0 fe80::cc3:b959:8f6b:fffd:546 :::*  

```

So at the moment looks like I have to go with switching between both modes hm… But if you got some ideas, just keep em coming, going to check back tomorrow 🙂

---

<div class="post-metadata">

**Author:** ![nmaas87](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/nmaas87/32/5570_2.png) [@nmaas87](https://forums.balena.io/u/nmaas87)\
**Post date:** [March 16, 2022, 7:45pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/15 "2022-03-16T19:45:34Z")

</div>

Ok, another issue: How to set HOST\_CONFIG settings?

I added multiple parameters which should be picked up by the supervisor and included in the config.txt - but nothing happens with them?

Any idea?

`````auto
version: '2'
services:
  specialsensor:
    #image: balenablocks/sensor
    build: .
    privileged: true
    environment:
      - MQTT_ADDRESS=x
      - MQTT_PUB_INTERVAL=10
      - MQTT_PUB_TOPIC=x
      - MQTT_USE_TLS=1
      - MQTT_PORT=8883
      - MQTT_USERNAME=x
      - MQTT_PASSWORD=1
      - COLLAPSE_FIELDS=1
      - RAW_VALUES=0
      - BALENA_HOST_CONFIG_dtoverlay="disable-bt"
      - RESIN_HOST_CONFIG_disable_splash=1
      - RESIN_HOST_CONFIG_gpu_mem=192
      - BALENA_HOST_CONFIG_hdmi_group=2
      - BALENA_HOST_CONFIG_hdmi_mode=85
      - BALENA_HOST_CONFIG_display_hdmi_rotate=1
      - BALENA_HOST_CONFIG_hdmi_force_hotplug=1
      - BALENA_HOST_CONFIG_enable_uart=1
    labels:
      io.balena.features.kernel-modules: '1'
      io.balena.features.sysfs: '1'
      io.balena.features.supervisor-api: '1'
    ````
`````

---

<div class="post-metadata">

**Author:** ![nmaas87](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/nmaas87/32/5570_2.png) [@nmaas87](https://forums.balena.io/u/nmaas87)\
**Post date:** [March 23, 2022, 5:48pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/16 "2022-03-23T17:48:17Z")

</div>

Well, I did solve it in that way that I set the config.txt to the correct values before first boot, so that I did not have to touch the sqlite database on the balenaOS.

I have now deployed two RPi 4 and have one additional RPi 4 and one RPi 3 (aarch64) already prepared and going to field-deploy tomorrow. Thats a good thing and I guess more will follow as soon as we got this RPi-Crisis over with. But the results from the field are already very impressive and I think the sensors I deployed show some irregularities we are currently already stretching our heads about, even with only two days of recorded data.

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [March 31, 2022, 2:58pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/17 "2022-03-31T14:58:05Z")

</div>

How is this running @nmaas87 ? sorry i didn’t get your pings here? ☹

---

<div class="post-metadata">

**Author:** ![nmaas87](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/nmaas87/32/5570_2.png) [@nmaas87](https://forums.balena.io/u/nmaas87)\
**Post date:** [March 31, 2022, 3:15pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/18 "2022-03-31T15:15:18Z")

</div>

Hey Marc, this did work out great so far. I got 3x RPi 4-64 and 1x RPi 3-64 with mixed workloads (all working with BME280, some with SGP30 for adding additional use to making lab spaces more secure and tech spaces more supervised - and running flight maps of the ISS or mission clocks in our controlroom). As soon as I get more RPis (currently backordered) this is going to increase. But yeah, so far so good. I am going to have to think about an update path / allow for ease of updating the balenaOS in an more offline fashion. Other than that, I am good. Loads better than the old design I threw in there a year ago with systems in developer mode. This was a horrible idea back then…

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [March 31, 2022, 6:05pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/19 "2022-03-31T18:05:00Z")

</div>

@nmaas87 did you try the [offline updates](https://www.balena.io/docs/learn/deploy/offline-updates/)?

I’m looking forward to get your point of view of the offline updates 🙂

---

<div class="post-metadata">

**Author:** ![nmaas87](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/nmaas87/32/5570_2.png) [@nmaas87](https://forums.balena.io/u/nmaas87)\
**Post date:** [March 31, 2022, 6:15pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/20 "2022-03-31T18:15:56Z")

</div>

@mpous  
I am not really sure if my understanding is correctly (or if the options have changed, because the website looks a bit different than the last time I visited :D) - but in the back of my mind I had the little voice that you’d need to push your code to balenaCloud first so that you could create an offline update / preload from it. Is that still the case? Because we cannot push this code to any online service, only locally work on e.g. our own laptop is allowed.

---

<div class="post-metadata">

**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\
**Post date:** [April 1, 2022, 4:55pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/21 "2022-04-01T16:55:19Z")

</div>

Indeed! instructions are here [Offline updates - Balena Documentation](https://www.balena.io/docs/learn/deploy/offline-updates/#overview-of-the-process)

Looking forward to get your feedback when you test this 😃

---

<div class="post-metadata">

**Author:** ![nmaas87](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/nmaas87/32/5570_2.png) [@nmaas87](https://forums.balena.io/u/nmaas87)\
**Post date:** [April 2, 2022, 4:38pm UTC](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526/22 "2022-04-02T16:38:05Z")

</div>

I did study the instructions but it is sadly still not viable for our usecase as it is necessary to push code to balenaCloud / deploy to balenaCloud for the offline update to work. With that I cannot use this technique but would resort to recreating SD like on initial deployment and swapping them out.

Sorry!

[Next page](https://forums.balena.io/t/using-balenaos-securely-production-image-offline/354526.md?page=2)