Upgrade 3.8.2 -> 4.1.0

openBalena
1

openBalena Upgrade - Guide (v3.8.2 ➜ v4.1.x+)

Status: FINAL / COMPLETE
Goal: Upgrade an existing openBalena 3.8.2 installation to a modern 4.1.x-compatible stack without losing data
Tested context: self-hosted openBalena with MinIO (S3), Registry v2, custom CA / self-signed certificates

0. FIRST STEPS (DO NOT SKIP)

  1. Create a full VM snapshot
  2. Verify restore works
  3. Strongly recommended: clone the server and test the upgrade there first

During the upgrade, always watch logs:

docker logs -f openbalena-api-1
docker logs -f openbalena-registry-1
docker logs -f openbalena-haproxy-1
docker logs -f openbalena-db-1

2. DNS – Existing Records and Additional Requirement**

The following DNS records should already exist in a working openBalena installation and must correctly resolve to the base domain.

Required DNS records

The following CNAME records are mandatory and must be created exactly as listed:

CNAME  api.staging.example.com        -> staging.example.com.
CNAME  registry.staging.example.com   -> staging.example.com.
CNAME  vpn.staging.example.com        -> staging.example.com.
CNAME  s3.staging.example.com         -> staging.example.com.
CNAME  tunnel.staging.example.com     -> staging.example.com.
CNAME  data.staging.example.com       -> staging.example.com.
CNAME  staging.example.com            -> staging.example.com.

As part of this upgrade, one additional DNS record must be present :

CNAME  cloudlink.staging.example.com  -> staging.example.com.

1. System prerequisites

sudo apt-get update
sudo apt-get install -y make openssl git jq

1.1 Stop containers

./open-balena/scripts/compose down

1.2 Switch to balena user & normalize project folder name

Docker Compose project name = folder name.

sudo su - balena
cd /home/balena
mv open-balena openbalena 2>/dev/null || true
cd openbalena

Clean local overrides if present:

git fetch --all --tags
git restore compose/version 2>/dev/null || true
git checkout v4.1.0

3. Set base domain (DNS_TLD)

export DNS_TLD="staging.example.com"

You may also persist this in .env:

SUPERUSER_EMAIL=""

4. Docker Compose – map volumes to OLD volumes (CRITICAL)

volumes:
  db-data:
    external: true
    name: openbalena_db

  s3-data:
    external: true
    name: openbalena_s3

This guarantees:

  • old Postgres data is reused
  • old MinIO are reused

5. Docker Compose – REQUIRED environment changes

In this section you adjust environment variables and service settings required for the 4.1.x stack.

5.2 S3 (MinIO) – force correct region & disable cache

services:
  s3:
    environment:
      COMMON_REGION: us-east-1
      REGISTRY2_CACHE_ENABLED: "false"

5.3 Registry – force region

services:
  registry:
    environment:
      COMMON_REGION: us-east-1

5.4 Prevent test containers from starting

profiles: ["test"]

Apply to:

  • dut
  • sut
  • tag-sidecar
  • docker

5.5 Remove Version Tag

version: ''

6. Fix Makefile wait target (required)

.PHONY: wait
wait: ## Wait for a service to become healthy
	@cid="$$(docker compose ps -q $(SERVICE) | head -n 1)"; \
	until [ -n "$$cid" ] && \
	      [ "$$(docker inspect -f '{{if .State.Health}}{{.State.Health.Status}}{{else}}nohealthcheck{{end}}' $$cid 2>/dev/null)" = "healthy" ]; do \
		printf '.'; \
		sleep 3; \
		cid="$$(docker compose ps -q $(SERVICE) | head -n 1)"; \
	done; \
	printf '\n'

7. HAProxy routing fixes

Apply the following rules in both http and https frontends.

acl host-registry-backend hdr_beg(host) -i "registry." "registry2."
use_backend registry-backend if host-registry-backend

Legacy config support (insert above all other acl rules):

acl legacy_config path -m reg -i ^/config(\.json)?$
http-request replace-path ^/config(\.json)?$ /os/v1/config if legacy_config

12. Certificates – Keep Existing CA (Recommended)

This section explains how to reuse an existing Certificate Authority (CA) when upgrading openBalena.
This is strongly recommended, especially if devices are already provisioned and pinned to the old CA.

12.1 Required .env Variables

You need three variables in your .env file:

  • HAPROXY_CRT – Server certificate including full chain (PEM, base64)
  • HAPROXY_KEY – Server private key (PEM, base64)
  • ROOT_CA – Root CA certificate (PEM, base64)

12.2 If You Already Have Valid Certificates (Recommended Path)

If your certificates are not expired and still trusted by your devices, you can reuse them directly.

From your old openBalena installation (/config/active):

New Variable Old Variable
HAPROXY_CRT OPENBALENA_ROOT_CRT
HAPROXY_KEY OPENBALENA_ROOT_KEY
ROOT_CA OPENBALENA_ROOT_CA

Simply copy these values into the new .env.

12.3 If Certificates Are Deprecated or Need Renewal

If certificates are expired, invalid, or the domain changed, generate a new HAProxy certificate signed by the existing Root CA.

Existing CA files

  • CA private key: /openbalena/config/certs/root/private/ca.key
  • Root CA certificate: root-ca.pem

12.4 Generate a New HAProxy Certificate

1) Generate server private key

openssl genrsa -out haproxy.key.pem 4096

2) Create CSR

openssl req -new   -key haproxy.key.pem   -out haproxy.csr.pem   -subj "/CN=staging.example.com"

3) Define SANs

cat > san.ext <<'EOF'
subjectAltName=DNS:*.staging.example.com,DNS:staging.example.com
basicConstraints=CA:FALSE
keyUsage=digitalSignature,keyEncipherment
extendedKeyUsage=serverAuth
EOF

4) Sign with existing Root CA

echo -n "$OPENBALENA_ROOT_CA" | base64 --decode > ca.pem
openssl x509 -req   -in haproxy.csr.pem   -CA ca.pem   -CAkey ca.key   -CAcreateserial   -out haproxy.crt.pem   -days 825   -sha256   -extfile san.ext

12.5 Verify Certificate

openssl x509   -in haproxy.crt.pem   -noout   -issuer   -subject   -dates   -ext subjectAltName

12.6 Convert to .env Format

HAPROXY_CRT=$(base64 < haproxy.crt.pem | tr -d '\n')
HAPROXY_KEY=$(base64 < haproxy.key.pem | tr -d '\n')
ROOT_CA=$(base64 < ca.pem | tr -d '\n')

12.7 Write to .env

echo "HAPROXY_CRT=$HAPROXY_CRT" >> .env
echo "HAPROXY_KEY=$HAPROXY_KEY" >> .env
echo "ROOT_CA=$ROOT_CA" >> .env

Restart HAProxy:

docker compose up -d
docker restart openbalena-haproxy-1

7. Start DB Container

Before starting the database container, make sure Docker Compose v2 is installed.
openBalena requires Docker Compose v2 for all make and docker compose commands.

Install Docker Compose v2 (per-user)

mkdir -p ~/.docker/cli-plugins

curl -SL \
  https://github.com/docker/compose/releases/download/v2.25.0/docker-compose-linux-x86_64 \
  -o ~/.docker/cli-plugins/docker-compose

chmod +x ~/.docker/cli-plugins/docker-compose

Verify the installation:

docker compose version

Expected output (example):

Docker Compose version v2.25.0

Start services

docker compose up -d

Start the database container

The database container must be running and healthy before continuing with the next upgrade steps.

8. DATABASE MIGRATION PROBLEMS (Postgres 14 ➜ 16)

8.1 API migration 0074 crash (MOST COMMON)

Option A – quick fix (delete migration file)

docker exec -it openbalena-api-1 sh -lc 'rm -f src/migrations/0074-normalize-release-contract.async.ts'
docker restart openbalena-api-1

:warning: Repeat after rebuilds.

Option B – clean fix (mark migration as executed)

docker stop openbalena-api-1
docker exec -it openbalena-db-1 psql -U docker -d resin -c \
"UPDATE \"migration\"
 SET \"executed migrations\" =
   (
     CASE
       WHEN (\"executed migrations\"::jsonb) @> '[\"0074\"]'::jsonb
         THEN \"executed migrations\"::jsonb
       ELSE (\"executed migrations\"::jsonb || '[\"0074\"]'::jsonb)
     END
   )::text
 WHERE \"model name\" = 'resin';"

Clear migration lock if needed:

docker exec -it openbalena-db-1 psql -U docker -d resin -c "
DELETE FROM "migration lock" WHERE "model name"='resin';
"

9. Fix boolean <> integer DB error

docker exec -it openbalena-db-1 psql -U docker -d resin

ALTER TABLE release DROP CONSTRAINT "release$69zgYrVSJaN1avGiEeipPlJ9/lMKzOIt3iMPF6u/6WY=";
ALTER TABLE release DROP CONSTRAINT "release$RcddhgkY+99IgKXAUId7Q3iN4WylzgAxSFiF+JvyRiY=";

Restart API:

docker compose down

Start services:

make up

13. balena CLI – TRUST SELF-SIGNED CA

On your laptop:

export NODE_EXTRA_CA_CERTS=/path/to/ca.pem

Login:

balena login --unsupported

DONE :tada: