RPi4 / balenaOS 2.82.10+rev1 self-restart and overwrite of config.txt

Product support
1

Dear all,
I am using the current balena-browser block and a nginx instance as a localy deployed app onto a balenaOS 2.82.10+rev1 dev RPi 4 / 8 GB machine. The problem is that the browser plugin does give me a lot of artefacts and errors. If I manually edit the config.txt and change gpu_mem from 16 to 192, this goes away, but some seconds after boot the RPi reboots and changes back the settings / recreating the errors (it also drops all other config.txt changes, like forced HDMI modes etc). I guess there is some watchdog within the balena-browser block or balena engine.
Do I need to add something like
BALENA_HOST_CONFIG_gpu_mem=192
to my docker-compose.yml … to make this change right or where do I have to add it on a local-dev machine / not connected to the internet?

Cheers,

Nico

2

Maybe @chrisys / @phil-d-wilson knows something?
I might be doing something wrong: I need to change config.txt items multiple times in offline, resinOS development machines (“local”, not added to balena.io, provisioned via the balena cli with belana push). How to do that? Is there anything to use the BALENA_HOST_CONFIG env variables with the offline balena cli client? And if not, which file do I have to change that the changes stay - and not are overwritten by an angry balena supervisor after next boot?

(Thats a big question, because a lot of more hardware driven balena blocks become useless in an airgapped environment if there is no option for that within balena cli :confused: )

Cheers
Nico

PS: balena push with --env “BALENA_HOST_CONFIG_gpu_mem=192” --env “BALENA_HOST_CONFIG_hdmi_group=1” --env “BALENA_HOST_CONFIG_hdmi_mode=16” --nocache did not work / did not get the browser block working, it is still in the wrong mode as well as showing a lot of screen tearing and errors as it only has 16 MB of GPU RAM.

3

That’s a good question and one I don’t immediately know the answer to! Will try to find out what you can do here.

1 Like
4

Thank you, very appreciated! :slight_smile:

5

Hi Nico,

That is indeed a great question, here is the explanation to the behavior you observe (followed by a workaround), you already might know some of this, but I’ll try to make it a generic answer, so please bear with me

  • The supervisor’s only function is to get the device from its current state into a target state (usually given by the cloud).
  • On first boot, the supervisor takes the current state of the device and stores it on its own database as the target while it gets a new target from the cloud.
  • If something changes locally and that doesn’t match the target, the supervisor will the state of the device to the stored target.
  • For unmanaged devices, as you point out, this doesn’t work so well, since they’ll never get a cloud target, any change will be reverted by the supervisor update

We still have to discuss what the proper solution should be, I’m leaning towards making this initial target volatile until the device can provision, but there might be other possibilities. In the meantime, the workaround is using the supervisor API, here is how you can do it

  • Get the current target state for the device and store it in a file
    curl -X GET http://<your-device-ip>:48484/v2/local/target-state | jq ".state" > state.json
  • Modify the target state adding the variables you want in the config section, e.g. BALENA_HOST_CONFIG_gpu_mem=192
  • Set a new target state for the supervisor
curl -X POST --header 'Content-type: application/json' -d @state.json http://<your-device-ip>:48484/v2/local/target-state

Hope this helps
Felipe

6

Dear pipex,

thanks for the information :slight_smile:
This is the state information I get.

{
  "local": {
    "name": "local",
    "config": {
      "SUPERVISOR_POLL_INTERVAL": "60000",
      "SUPERVISOR_INSTANT_UPDATE_TRIGGER": "true",
      "SUPERVISOR_LOCAL_MODE": "true",
      "SUPERVISOR_CONNECTIVITY_CHECK": "true",
      "SUPERVISOR_LOG_CONTROL": "true",
      "SUPERVISOR_DELTA": "false",
      "SUPERVISOR_DELTA_REQUEST_TIMEOUT": "30000",
      "SUPERVISOR_DELTA_APPLY_TIMEOUT": "0",
      "SUPERVISOR_DELTA_RETRY_COUNT": "30",
      "SUPERVISOR_DELTA_RETRY_INTERVAL": "10000",
      "SUPERVISOR_DELTA_VERSION": "2",
      "SUPERVISOR_OVERRIDE_LOCK": "false",
      "SUPERVISOR_PERSISTENT_LOGGING": "false",
      "HOST_FIREWALL_MODE": "",
      "HOST_DISCOVERABILITY": "true",
      "SUPERVISOR_HARDWARE_METRICS": "true",
      "SUPERVISOR_VPN_CONTROL": "false",
      "HOST_CONFIG_gpu_mem": "16",
      "HOST_CONFIG_enable_uart": "1",
      "HOST_CONFIG_dtoverlay": "\"vc4-fkms-v3d\"",
      "HOST_CONFIG_arm_64bit": "1",
      "HOST_CONFIG_dtparam": "\"i2c_arm=on\",\"spi=on\",\"audio=on\"",
      "HOST_CONFIG_disable_splash": "1"
    },
    "apps": {
      "1": {
        "appId": 1,
        "commit": "localrelease",
        "releaseId": 1,
        "services": [
          {
            "imageId": 1,
            "serviceName": "browser",
            "appId": 1,
            "releaseId": 1,
            "serviceId": 1,
            "imageName": "local_image_browser:latest",
            "dependsOn": null,
            "config": {
              "environment": {
                "KIOSK": "1",
                "SHOW_CURSOR": "0",
                "ENABLE_GPU": "1",
                "BALENA_APP_ID": "1",
                "BALENA_APP_NAME": "localapp",
                "BALENA_SERVICE_NAME": "browser",
                "BALENA_DEVICE_UUID": "758adc960cf7db7fe8c4c6efab134f91",
                "BALENA_DEVICE_TYPE": "raspberrypi4-64",
                "BALENA_DEVICE_ARCH": "aarch64",
                "BALENA_HOST_OS_VERSION": "balenaOS 2.82.10+rev1",
                "BALENA_APP_LOCK_PATH": "/tmp/balena/updates.lock",
                "BALENA": "1",
                "RESIN_APP_ID": "1",
                "RESIN_APP_NAME": "localapp",
                "RESIN_SERVICE_NAME": "browser",
                "RESIN_DEVICE_UUID": "758adc960cf7db7fe8c4c6efab134f91",
                "RESIN_DEVICE_TYPE": "raspberrypi4-64",
                "RESIN_DEVICE_ARCH": "aarch64",
                "RESIN_HOST_OS_VERSION": "balenaOS 2.82.10+rev1",
                "RESIN_APP_LOCK_PATH": "/tmp/balena/updates.lock",
                "RESIN": "1",
                "RESIN_SERVICE_KILL_ME_PATH": "/tmp/balena/handover-complete",
                "BALENA_SERVICE_HANDOVER_COMPLETE_PATH": "/tmp/balena/handover-complete",
                "USER": "root",
                "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LC_ALL": "C.UTF-8",
                "DEBIAN_FRONTEND": "noninteractive",
                "UDEV": "1",
                "QEMU_CPU": "arm1176",
                "PULSE_SERVER": "tcp:audio:4317"
              },
              "labels": {
                "io.balena.supervised": "true",
                "io.balena.app-id": "1",
                "io.balena.service-id": "1",
                "io.balena.service-name": "browser",
                "io.balena.architecture": "rpi",
                "io.balena.device-type": "raspberrypi",
                "io.balena.qemu.version": ""
              },
              "restart": "always",
              "networkMode": "host",
              "privileged": true,
              "volumes": [
                "1_settings:/data",
                "/tmp/balena-supervisor/services/1/browser:/tmp/resin",
                "/tmp/balena-supervisor/services/1/browser:/tmp/balena"
              ],
              "image": "sha256:fa7ca3bc2ac4314d5f68595bb8cb219c1b9259cffa39ae6f4617f1024543104c",
              "running": true,
              "hostname": "balena",
              "command": [
                "export DISPLAY=:0"
              ],
              "entrypoint": [
                "bash",
                "start.sh"
              ],
              "stopSignal": "SIGTERM",
              "workingDir": "/usr/src/app",
              "user": "",
              "oomKillDisable": false,
              "readOnly": false,
              "sysctls": {},
              "portMaps": [],
              "capAdd": [],
              "capDrop": [],
              "cgroupParent": "",
              "devices": [],
              "deviceRequests": [],
              "dnsOpt": [],
              "extraHosts": [],
              "expose": [],
              "networks": {},
              "dns": [],
              "dnsSearch": [],
              "ulimits": {},
              "groupAdd": [],
              "healthcheck": {
                "test": [
                  "NONE"
                ]
              },
              "pid": "",
              "pidsLimit": 0,
              "securityOpt": [],
              "stopGracePeriod": 10,
              "tmpfs": [],
              "usernsMode": "",
              "cpuShares": 0,
              "cpuQuota": 0,
              "cpus": 0,
              "cpuset": "",
              "domainname": "",
              "ipc": "shareable",
              "macAddress": "",
              "memLimit": 0,
              "memReservation": 0,
              "oomScoreAdj": 0,
              "shmSize": 67108864,
              "tty": true
            }
          },
          {
            "imageId": 2,
            "serviceName": "nginx",
            "appId": 1,
            "releaseId": 1,
            "serviceId": 2,
            "imageName": "local_image_nginx:latest",
            "dependsOn": null,
            "config": {
              "environment": {
                "BALENA_APP_ID": "1",
                "BALENA_APP_NAME": "localapp",
                "BALENA_SERVICE_NAME": "nginx",
                "BALENA_DEVICE_UUID": "758adc960cf7db7fe8c4c6efab134f91",
                "BALENA_DEVICE_TYPE": "raspberrypi4-64",
                "BALENA_DEVICE_ARCH": "aarch64",
                "BALENA_HOST_OS_VERSION": "balenaOS 2.82.10+rev1",
                "BALENA_APP_LOCK_PATH": "/tmp/balena/updates.lock",
                "BALENA": "1",
                "RESIN_APP_ID": "1",
                "RESIN_APP_NAME": "localapp",
                "RESIN_SERVICE_NAME": "nginx",
                "RESIN_DEVICE_UUID": "758adc960cf7db7fe8c4c6efab134f91",
                "RESIN_DEVICE_TYPE": "raspberrypi4-64",
                "RESIN_DEVICE_ARCH": "aarch64",
                "RESIN_HOST_OS_VERSION": "balenaOS 2.82.10+rev1",
                "RESIN_APP_LOCK_PATH": "/tmp/balena/updates.lock",
                "RESIN": "1",
                "RESIN_SERVICE_KILL_ME_PATH": "/tmp/balena/handover-complete",
                "BALENA_SERVICE_HANDOVER_COMPLETE_PATH": "/tmp/balena/handover-complete",
                "USER": "root",
                "PATH": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "LC_ALL": "C.UTF-8",
                "DEBIAN_FRONTEND": "noninteractive",
                "UDEV": "off"
              },
              "labels": {
                "io.balena.local.image": "1",
                "io.balena.local.service": "nginx",
                "io.balena.supervised": "true",
                "io.balena.app-id": "1",
                "io.balena.service-id": "2",
                "io.balena.service-name": "nginx",
                "io.balena.architecture": "aarch64",
                "io.balena.device-type": "jetson-tx2",
                "io.balena.qemu.version": "4.0.0+balena2-aarch64"
              },
              "restart": "always",
              "networkMode": "host",
              "image": "sha256:59ce32259138fd949fa4eae1db58c7b9a46e9d352a3ffb5a965372df284dbc85",
              "running": true,
              "hostname": "balena",
              "command": [
                "nginx",
                "-g",
                "daemon off;"
              ],
              "entrypoint": [
                "/usr/bin/entry.sh"
              ],
              "stopSignal": "SIGTERM",
              "workingDir": "/usr/src/app",
              "user": "",
              "volumes": [
                "/tmp/balena-supervisor/services/1/nginx:/tmp/resin",
                "/tmp/balena-supervisor/services/1/nginx:/tmp/balena"
              ],
              "oomKillDisable": false,
              "readOnly": false,
              "sysctls": {},
              "portMaps": [
                {
                  "ports": {
                    "internalStart": 80,
                    "internalEnd": 80,
                    "externalStart": 80,
                    "externalEnd": 80,
                    "host": "",
                    "protocol": "tcp"
                  }
                }
              ],
              "capAdd": [],
              "capDrop": [],
              "cgroupParent": "",
              "devices": [],
              "deviceRequests": [],
              "dnsOpt": [],
              "extraHosts": [],
              "expose": [
                "80/tcp"
              ],
              "networks": {},
              "dns": [],
              "dnsSearch": [],
              "ulimits": {},
              "groupAdd": [],
              "healthcheck": {
                "test": [
                  "NONE"
                ]
              },
              "pid": "",
              "pidsLimit": 0,
              "securityOpt": [],
              "stopGracePeriod": 10,
              "tmpfs": [],
              "usernsMode": "",
              "cpuShares": 0,
              "cpuQuota": 0,
              "cpus": 0,
              "cpuset": "",
              "domainname": "",
              "ipc": "shareable",
              "macAddress": "",
              "memLimit": 0,
              "memReservation": 0,
              "oomScoreAdj": 0,
              "privileged": false,
              "shmSize": 67108864,
              "tty": true
            }
          }
        ],
        "volumes": {
          "settings": {
            "name": "settings",
            "appId": 1,
            "config": {
              "driverOpts": {},
              "driver": "local",
              "labels": {
                "io.balena.supervised": "true"
              }
            }
          }
        },
        "networks": {
          "default": {
            "name": "default",
            "appId": 1,
            "config": {
              "driver": "bridge",
              "ipam": {
                "driver": "default",
                "config": [],
                "options": {}
              },
              "enableIPv6": false,
              "internal": false,
              "labels": {},
              "options": {}
            }
          }
        }
      }
    }
  },
  "dependent": {
    "apps": [],
    "devices": []
  }
}

Trying to push it back to the RPi 4 without changing anything gives following error:

{"status":"failed","message":"Invalid apps"}
  • and I really don’t know why.

From the structure I can see one needs to add the changes to the local, config tree and substitute the starting parameter, so out of

 "BALENA_HOST_CONFIG_gpu_mem": "192",

should become

 "HOST_CONFIG_gpu_mem": "192",

is that correct?
If you got some new infos and I can iterate on them thanks for your help!

While this thread shall be about resolving the issue above, I got to think of a new usecase for the balena environment and how to make it suitable for extremly enclosed environments, so I am adding
@mpous @dtischler and @ajlennon - maybe the last one can support this idea and the first ones decide that the idea is not to stupid and worthwhile :stuck_out_tongue:

To be upfront: I know running a device locally without any external “cloud backend” is not really the main goal of balena, however - there is a niche (especially in the aerospace sector and other industrial applications) where either complete airgaps exist, connectivity is limited, data is classified or a ban on usage of cloud environments at all exist. However, there are still dozens of applications were balena would still be useable, if you were able to provision a new device via a special laptop (which is already possible with balena-cli) from a local codebase, which is not send to the cloud (thats balena push ) and put onto the device. Then the device still can be scanned and then put into production.

However, there are still two things to be needed for that
a) support of the device variables for a local deployment. These device variables should be kept within textfiles, so that the really complete fleet/application can be kept within one git repo and be provisioned as needed without too many manual steps. While docker-compose.yml would be a good idea, I think you started using these device variables already in the balena.yml for the balenaHub applications - and so it would make perfect sense to configure a local devices variables (e.g. gpu_mem etc) by using balena push from a balena.yml in the folder [ gpsTime/balena.yml at main · nmaas87/gpsTime · GitHub data / applicationConfigVariables ].

b) there needs to be an option to harden the device, lets just name it protected local mode. within the dev local mode, the device is really open. with e.g. supervisor answering questions and getting new configs, ssh via 22222 / root, etc. It would be a great idea for a usage as stated above if you could provision a such hardend device on the first connection with a certificate and close all connection other than via ssh on 22222 using this certificate. Meaning every other port and the local terminal should be closed and only be reachable via this cert and secured connection. balena-cli could still work its functions, if it were to use the cert and tunnel via ssh, but the device would not be easyily attackable (this is also in regards of these environments, as nessus scans for open ports and such are a thing and then the test which can be done with the device using these open ports).

Cheers

Nico

1 Like
7

Ok, short interlude to an extremly messy but working solution:

  • Copy the sqlite database to your computer
scp -P22222 root@<IP>:/mnt/data/resin-data/balena-supervisor/database.sqlite .
  • Open it with e.g. SqliteBrowser: https://sqlitebrowser.org/
  • Navigate to deviceConfig, targetValues and change what you need there, e.g. overwrite the old HOST_CONFIG_gpu_mem:16 with your values, e.g. “HOST_CONFIG_gpu_mem”:“192”,“HOST_CONFIG_hdmi_group”:“1”,“HOST_CONFIG_hdmi_mode”:“16”,
  • Apply changes and save the file
  • Upload and overwrite the old sqlite database
scp -P22222 database.sqlite root@<IP>:/mnt/data/resin-data/balena-supervisor/database.sqlite
  • Just for good measure, reconnect via ssh and sync the filesystem
ssh <IP> -p22222 -lroot
sync
  • Wait a second or two and pull the powerplug, then reconnect. The RPi will reboot with the “wrong” settings, start the supervisor, realize its wrong, change the parameters, reboot and then will keep them.

Yes, you should never overwrite a running and used sqlite database like this.
Yes, it can break your system.
No, thats on you.
Yes, its a hack but it works :stuck_out_tongue: (and you know I am a very pragmatic person… so sorry for that :stuck_out_tongue: )

If one of the balena ones got a better hack, please let me know and also - having this as a real solution (e.g. via balena.yml) would be extremly favorable.

1 Like
8

And another update, if you need to make it easier, you can go the linux route / without gui:

  • Just install sqlite3 onto your system:
sudo apt update
sudo apt install -y sqlite3
  • Then get the database.sqlite like written above:
cd ~
scp -P22222 root@<IP>:/mnt/data/resin-data/balena-supervisor/database.sqlite .
  • extract the targetValues as json file from the database:
sqlite3 database.sqlite "SELECT targetValues FROM deviceConfig;" > targetValues.json
  • make your changes to the json file and write it back to the database
sqlite3 database.sqlite "UPDATE deviceConfig SET targetValues='`cat targetValues.json`';"
  • then upload the database again
scp -P22222 database.sqlite root@<IP>:/mnt/data/resin-data/balena-supervisor/database.sqlite
  • sync
ssh <IP> -p22222 -lroot sync
  • and powercycle your device

Great if you need to provision more than one device and want to prepare a json file with things to add to all devices :wink:

1 Like
9

Hi Nico,

  • If you want to interact with the supervisor database you can also use these instructions
  • Regarding the Invalid apps error. I see the problem, i just realised the GET /v2/local/target-state is returning services as an array instead of an object keyed by serviceId which is what the supervisor is expecting. It is also returning some properties with the wrong name. That is a bug although we might need some time to figure out how to proceed, since some users might be relying on this format.

This means that for now, the database might be the only workaround to change config values in unmanaged mode :frowning:

I will bring up allowing the CLI to set config variables on our internal discussion forum to see if we get any traction.

  • Regarding securing the device, we do have plans (and some work in progress) to unify the OS variants and make both dev and prod variants be secure. There are some things to figure out still but some of the ideas go in the lines of what you commented, accessing the docker socket and API through a ssh socket is one idea.

Thanks again for your feedback, this was actually very enlightening for me as well :slight_smile:

1 Like
10

Thank you Felipe,

and sorry for making a mess. I am always using devices in not-intended or unusual ways, so - sorry :slight_smile: . But thanks for the feedback, don’t worry about the API problems - good to hear that they are already worked upon.

Securing the devices is also a good point and thanks for taking this up as well. For the moment I can work with what I know now - thanks :slight_smile:

12

Wow you have been busy @nmaas87 !!!

1 Like
14

Yeah sorry, had to meet a personal deadline for some offline running Wallclock within a webrowser, just to tell time within it for… control room reasons ;). Its not deployed yet, but the prototype works, needs to be streamlined and maybe the gals and guys here make my hacky-stuff a bit more… professional :wink:

2 Likes