I have been trying to get OpenBalena work on an EC2 instance that is using an ARM64 architecture.
The basic quickstart guide does not work with that, as containers just won’t start. Using the following compose/versions file:
export OPENBALENA_API_VERSION_TAG=v20.2.4
export OPENBALENA_DB_VERSION_TAG=v5.2.2
export OPENBALENA_MDNS_PUBLISHER_VERSION_TAG=v1.27.99
export OPENBALENA_REGISTRY_VERSION_TAG=v2.39.45
export OPENBALENA_S3_VERSION_TAG=v2.28.32
export OPENBALENA_VPN_VERSION_TAG=v11.29.36
All containers are able to start, but the api does not seem to be starting.
I used ./scripts/compose exec api journalctl -fn100 and I get the following output:
[balena@ip-10-76-5-22 open-balena]$ ./scripts/compose exec api journalctl -fn100
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[357]: grep: /balena/.env: No such file or directory
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[365]: grep: /balena/.env: No such file or directory
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[373]: grep: /balena/.env: No such file or directory
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[387]: grep: /balena/.env: No such file or directory
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[407]: grep: /balena/.env: No such file or directory
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[414]: grep: /balena/.env: No such file or directory
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[421]: grep: /balena/.env: No such file or directory
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[428]: grep: /balena/.env: No such file or directory
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[435]: grep: /balena/.env: No such file or directory
Apr 08 18:22:51 06eb1392ea33 avahi-daemon[63]: Server startup complete. Host name is 06eb1392ea33.local. Local service cookie is 1822443482.
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[448]: grep: /balena/.env: No such file or directory
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[455]: grep: /balena/.env: No such file or directory
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[462]: grep: /balena/.env: No such file or directory
Apr 08 18:22:51 06eb1392ea33 configure-balena.sh[469]: grep: /balena/.env: No such file or directory
Apr 08 18:22:52 06eb1392ea33 configure-balena.sh[476]: grep: /balena/.env: No such file or directory
Apr 08 18:22:52 06eb1392ea33 configure-balena.sh[483]: grep: /balena/.env: No such file or directory
Apr 08 18:22:52 06eb1392ea33 configure-balena.sh[64]: Installing custom CA bundle...
Apr 08 18:22:52 06eb1392ea33 configure-balena.sh[490]: Updating certificates in /etc/ssl/certs...
Apr 08 18:22:54 06eb1392ea33 configure-balena.sh[1490]: rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
Apr 08 18:22:54 06eb1392ea33 configure-balena.sh[490]: 1 added, 0 removed; done.
Apr 08 18:22:54 06eb1392ea33 configure-balena.sh[490]: Running hooks in /etc/ca-certificates/update.d...
Apr 08 18:22:54 06eb1392ea33 configure-balena.sh[490]: done.
Apr 08 18:22:54 06eb1392ea33 configure-balena.sh[1499]: 2024-04-08T18:22:54Z 06eb1392ea33 /usr/local/bin/confd[1499]: INFO Backend set to env
Apr 08 18:22:54 06eb1392ea33 configure-balena.sh[1499]: 2024-04-08T18:22:54Z 06eb1392ea33 /usr/local/bin/confd[1499]: INFO Starting confd
Apr 08 18:22:54 06eb1392ea33 configure-balena.sh[1499]: 2024-04-08T18:22:54Z 06eb1392ea33 /usr/local/bin/confd[1499]: INFO Backend source(s) set to
Apr 08 18:22:54 06eb1392ea33 configure-balena.sh[1499]: 2024-04-08T18:22:54Z 06eb1392ea33 /usr/local/bin/confd[1499]: INFO Target config /etc/ssl/private/cloudfront-pk.pem out of sync
Apr 08 18:22:54 06eb1392ea33 configure-balena.sh[1499]: 2024-04-08T18:22:54Z 06eb1392ea33 /usr/local/bin/confd[1499]: INFO Target config /etc/ssl/private/cloudfront-pk.pem has been updated
Apr 08 18:22:54 06eb1392ea33 configure-balena.sh[1499]: 2024-04-08T18:22:54Z 06eb1392ea33 /usr/local/bin/confd[1499]: INFO Target config /usr/src/app/config/env out of sync
Apr 08 18:22:54 06eb1392ea33 configure-balena.sh[1499]: 2024-04-08T18:22:54Z 06eb1392ea33 /usr/local/bin/confd[1499]: INFO Target config /usr/src/app/config/env has been updated
Apr 08 18:22:54 06eb1392ea33 systemd[1]: Reloading.
Apr 08 18:22:54 06eb1392ea33 systemd[1]: /etc/systemd/system/open-balena-api.service:14: Unknown key name 'StartLimitIntervalSec' in section 'Service', ignoring.
Apr 08 18:22:54 06eb1392ea33 systemd[1]: confd.service: Deactivated successfully.
Apr 08 18:22:54 06eb1392ea33 systemd[1]: Finished confd.service - Confd.
Apr 08 18:22:54 06eb1392ea33 systemd[1]: Started open-balena-api.service - open-balena-api.
Apr 08 18:22:54 06eb1392ea33 systemd[1]: Starting rsyslog.service - System Logging Service...
Apr 08 18:22:54 06eb1392ea33 systemd[1]: rsyslog.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 18:22:54 06eb1392ea33 systemd[1]: rsyslog.service: Failed with result 'exit-code'.
Apr 08 18:22:54 06eb1392ea33 systemd[1]: Failed to start rsyslog.service - System Logging Service.
Apr 08 18:22:54 06eb1392ea33 systemd[1]: Reached target multi-user.target - Multi-User System.
Apr 08 18:22:54 06eb1392ea33 systemd[1]: Starting systemd-update-utmp-runlevel.service - Record Runlevel Change in UTMP...
Apr 08 18:22:54 06eb1392ea33 systemd[1]: systemd-update-utmp-runlevel.service: Deactivated successfully.
Apr 08 18:22:54 06eb1392ea33 systemd[1]: Finished systemd-update-utmp-runlevel.service - Record Runlevel Change in UTMP.
Apr 08 18:22:54 06eb1392ea33 systemd[1]: Startup finished in 4.833s.
Apr 08 18:22:55 06eb1392ea33 api[1522]: Running node-supervisor with
Apr 08 18:22:55 06eb1392ea33 api[1522]: program 'index.js'
Apr 08 18:22:55 06eb1392ea33 api[1522]: --watch 'src'
Apr 08 18:22:55 06eb1392ea33 api[1522]: --extensions 'js,node,coffee,sbvr,json,sql,pegjs,ts'
Apr 08 18:22:55 06eb1392ea33 api[1522]: --exec 'node'
Apr 08 18:22:55 06eb1392ea33 api[1522]: Starting child process with 'node index.js'
Apr 08 18:22:55 06eb1392ea33 api[1522]: Watching directory '/usr/src/app/src' for changes.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 1.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: Stopped rsyslog.service - System Logging Service.
Apr 08 18:22:55 06eb1392ea33 api[1522]: Press rs for restarting the process.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: Starting rsyslog.service - System Logging Service...
Apr 08 18:22:55 06eb1392ea33 systemd[1]: rsyslog.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 18:22:55 06eb1392ea33 systemd[1]: rsyslog.service: Failed with result 'exit-code'.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: Failed to start rsyslog.service - System Logging Service.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 2.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: Stopped rsyslog.service - System Logging Service.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: Starting rsyslog.service - System Logging Service...
Apr 08 18:22:55 06eb1392ea33 systemd[1]: rsyslog.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 18:22:55 06eb1392ea33 systemd[1]: rsyslog.service: Failed with result 'exit-code'.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: Failed to start rsyslog.service - System Logging Service.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 3.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: Stopped rsyslog.service - System Logging Service.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: Starting rsyslog.service - System Logging Service...
Apr 08 18:22:55 06eb1392ea33 systemd[1]: rsyslog.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 18:22:55 06eb1392ea33 systemd[1]: rsyslog.service: Failed with result 'exit-code'.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: Failed to start rsyslog.service - System Logging Service.
Apr 08 18:22:55 06eb1392ea33 api[1531]: Error: Cannot find module '/usr/src/app/init.js' imported from /usr/src/app/index.js
Apr 08 18:22:55 06eb1392ea33 api[1531]: Did you mean to import "./init.ts"?
Apr 08 18:22:55 06eb1392ea33 api[1531]: at finalizeResolution (node:internal/modules/esm/resolve:264:11)
Apr 08 18:22:55 06eb1392ea33 api[1531]: at moduleResolve (node:internal/modules/esm/resolve:924:10)
Apr 08 18:22:55 06eb1392ea33 api[1531]: at defaultResolve (node:internal/modules/esm/resolve:1148:11)
Apr 08 18:22:55 06eb1392ea33 api[1531]: at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:390:12)
Apr 08 18:22:55 06eb1392ea33 api[1531]: at ModuleLoader.resolve (node:internal/modules/esm/loader:359:25)
Apr 08 18:22:55 06eb1392ea33 api[1531]: at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:234:38)
Apr 08 18:22:55 06eb1392ea33 api[1531]: at ModuleLoader.import (node:internal/modules/esm/loader:322:34)
Apr 08 18:22:55 06eb1392ea33 api[1531]: at importModuleDynamically (node:internal/modules/esm/translators:160:35)
Apr 08 18:22:55 06eb1392ea33 api[1531]: at importModuleDynamicallyCallback (node:internal/modules/esm/utils:225:14)
Apr 08 18:22:55 06eb1392ea33 api[1531]: at start (file:///usr/src/app/index.js:8:2) {
Apr 08 18:22:55 06eb1392ea33 api[1531]: code: 'ERR_MODULE_NOT_FOUND',
Apr 08 18:22:55 06eb1392ea33 api[1531]: url: 'file:///usr/src/app/init.js'
Apr 08 18:22:55 06eb1392ea33 api[1531]: }
Apr 08 18:22:55 06eb1392ea33 api[1522]: Program node index.js exited with code 1
Apr 08 18:22:55 06eb1392ea33 systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 4.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: Stopped rsyslog.service - System Logging Service.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: Starting rsyslog.service - System Logging Service...
Apr 08 18:22:55 06eb1392ea33 systemd[1]: rsyslog.service: Main process exited, code=exited, status=1/FAILURE
Apr 08 18:22:55 06eb1392ea33 systemd[1]: rsyslog.service: Failed with result 'exit-code'.
Apr 08 18:22:55 06eb1392ea33 systemd[1]: Failed to start rsyslog.service - System Logging Service.
Apr 08 18:22:56 06eb1392ea33 systemd[1]: rsyslog.service: Scheduled restart job, restart counter is at 5.
Apr 08 18:22:56 06eb1392ea33 systemd[1]: Stopped rsyslog.service - System Logging Service.
Apr 08 18:22:56 06eb1392ea33 systemd[1]: rsyslog.service: Start request repeated too quickly.
Apr 08 18:22:56 06eb1392ea33 systemd[1]: rsyslog.service: Failed with result 'exit-code'.
Apr 08 18:22:56 06eb1392ea33 systemd[1]: Failed to start rsyslog.service - System Logging Service.
Apr 08 18:22:56 06eb1392ea33 systemd[1]: syslog.socket: Failed with result 'service-start-limit-hit'.
Apr 08 18:38:03 06eb1392ea33 systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Apr 08 18:38:03 06eb1392ea33 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Apr 08 18:38:03 06eb1392ea33 systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Here are the docker compose logs:
openbalena-cert-provider-1: [Error]e(B ACTIVE variable is not enabled. Value should be "true" or "yes" to continue.
openbalena-cert-provider-1: [Error]e(B Unable to continue due to misconfiguration. See errors above. [Stopping]
openbalena-db-1:
openbalena-db-1: PostgreSQL Database directory appears to contain a database; Skipping initialization
openbalena-db-1:
openbalena-db-1: 2024-04-08 18:22:50.010 UTC [1] LOG: starting PostgreSQL 16.1 (Debian 16.1-1.pgdg120+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
openbalena-db-1: 2024-04-08 18:22:50.011 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
openbalena-db-1: 2024-04-08 18:22:50.011 UTC [1] LOG: listening on IPv6 address "::", port 5432
openbalena-db-1: 2024-04-08 18:22:50.020 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
openbalena-db-1: 2024-04-08 18:22:50.031 UTC [31] LOG: database system was shut down at 2024-04-08 18:22:06 UTC
openbalena-db-1: 2024-04-08 18:22:50.127 UTC [1] LOG: database system is ready to accept connections
openbalena-db-1: 2024-04-08 18:27:50.127 UTC [29] LOG: checkpoint starting: time
openbalena-db-1: 2024-04-08 18:27:50.274 UTC [29] LOG: checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.134 s, sync=0.004 s, total=0.147 s; sync files=2, longest=0.003 s, average=0.002 s; distance=0 kB, estimate=0 kB; lsn=0/1D3B470, redo lsn=0/1D3B438
openbalena-redis-1: 1:C 08 Apr 2024 18:22:49.575 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
openbalena-redis-1: 1:C 08 Apr 2024 18:22:49.577 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
openbalena-redis-1: 1:C 08 Apr 2024 18:22:49.577 * Redis version=7.2.4, bits=64, commit=00000000, modified=0, pid=1, just started
openbalena-redis-1: 1:C 08 Apr 2024 18:22:49.577 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
openbalena-redis-1: 1:M 08 Apr 2024 18:22:49.578 * monotonic clock: POSIX clock_gettime
openbalena-redis-1: 1:M 08 Apr 2024 18:22:49.580 * Running mode=standalone, port=6379.
openbalena-redis-1: 1:M 08 Apr 2024 18:22:49.581 * Server initialized
openbalena-redis-1: 1:M 08 Apr 2024 18:22:49.584 * Loading RDB produced by version 7.2.4
openbalena-redis-1: 1:M 08 Apr 2024 18:22:49.584 * RDB age 43 seconds
openbalena-redis-1: 1:M 08 Apr 2024 18:22:49.584 * RDB memory usage when created 0.86 Mb
openbalena-redis-1: 1:M 08 Apr 2024 18:22:49.584 * Done loading RDB, keys loaded: 0, keys expired: 0.
openbalena-redis-1: 1:M 08 Apr 2024 18:22:49.584 * DB loaded from disk: 0.004 seconds
openbalena-redis-1: 1:M 08 Apr 2024 18:22:49.584 * Ready to accept connections tcp
openbalena-registry-1: Systemd init system enabled.
openbalena-vpn-1: Systemd init system enabled.
openbalena-api-1: Systemd init system enabled.
openbalena-haproxy-1: Building certificate from environment variables...
openbalena-haproxy-1: Setting up watches. Beware: since -r was given, this may take a while!
openbalena-haproxy-1: Watches established.
openbalena-haproxy-1: [NOTICE] 098/182251 (16) : New worker #1 (18) forked
openbalena-haproxy-1: [WARNING] 098/182251 (18) : Server backend_api/balena_api_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
openbalena-haproxy-1: [ALERT] 098/182251 (18) : backend 'backend_api' has no server available!
openbalena-haproxy-1: [WARNING] 098/182251 (18) : Server backend_registry/balena_registry_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
openbalena-haproxy-1: [ALERT] 098/182251 (18) : backend 'backend_registry' has no server available!
openbalena-haproxy-1: [WARNING] 098/182251 (18) : Server backend_vpn/balena_vpn_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
openbalena-haproxy-1: [ALERT] 098/182251 (18) : backend 'backend_vpn' has no server available!
openbalena-haproxy-1: [WARNING] 098/182252 (18) : Server backend_s3/balena_s3_1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
openbalena-haproxy-1: [ALERT] 098/182252 (18) : backend 'backend_s3' has no server available!
openbalena-haproxy-1: [WARNING] 098/182252 (18) : Server vpn-tunnel/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
openbalena-haproxy-1: [ALERT] 098/182252 (18) : proxy 'vpn-tunnel' has no server available!
openbalena-haproxy-1: [WARNING] 098/182253 (18) : Server vpn-tunnel-tls/balena_vpn is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
openbalena-haproxy-1: [ALERT] 098/182253 (18) : proxy 'vpn-tunnel-tls' has no server available!
openbalena-haproxy-1: [WARNING] 098/182256 (18) : Server backend_s3/balena_s3_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
openbalena-haproxy-1: [WARNING] 098/182257 (18) : Server backend_registry/balena_registry_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
openbalena-haproxy-1: [WARNING] 098/182259 (18) : Server backend_vpn/balena_vpn_1 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
openbalena-s3-1: Systemd init system enabled.
What do I need to do to get openBalena up and running? I’m not sure what exactly I need to do.
Should I just use an x86 instance instead of an arm64 instance? Is there going to be support for ARM down the line?
Thanks for the help!