Hi @brownster, we also needed this functionality, so we decided to build it out ourselves. We have plans to release our custom balenaos as open source once we get around to genericizing it sufficiently, but for now Iām happy to share the two components that enabled this for us.
First off, we had to modify the OS build process to use the balena jenkins scripts. Itās a pretty involved build process, but the key that is needed for OTA OS updates is the part where the build scripts deploy the OS to openbalena, similar to an app. To do this you first need to create a user in your openbalena instance called ābalena_osā and then create an application for each new device type you want to deploy an OS for, calling them ābalena_os/ā. You also need to add that custom device type (and associated alias) to your openbalena instance. To do all of this you will need to use a tool such as open-balena-admin or I suppose you can manually just add them in the database as well, because there is no out of the box tool available to manage users or custom device types in openbalena.
Once you have this, you then need your own wrapper script to slightly tweak the balena barys build script. Below is our example, yours may differ depending on how you want to manage your update process (i.e. do you want to use timestamp based updates?). By using this process it also allows you to tag certain devices to certain OS versions.
Build script wrapper (we call it harmoni_jenkins_build.sh):
#!/bin/bash
WORKSPACE="$(pwd)"
VARIANT="prod"
JENKINS_ARGUMENTS_VAR=""
export deploy=yes
# Process script arguments
args_number="$#"
while [[ $# -ge 1 ]]; do
arg=$1
case $arg in
-m|--machine)
if [ -z "$2" ]; then
echo "-m|--machine argument needs a machine name"
exit 1
fi
MACHINE="$2"
;;
-r|--rootdir)
if [ -z "$2" ]; then
echo "--rootdir needs directory name (root directory of the yocto project)"
exit 1
fi
WORKSPACE="${WORKSPACE:-$2}"
shift
;;
-d|--dev)
VARIANT="dev"
;;
-b|--build-only)
export deploy=no
;;
--preserve-build)
JENKINS_ARGUMENTS_VAR="$JENKINS_ARGUMENTS_VAR $1"
;;
--preserve-container)
JENKINS_ARGUMENTS_VAR="$JENKINS_ARGUMENTS_VAR $1"
;;
esac
shift
done
if [ -z "$MACHINE" ]; then
echo -e "[ERROR] $0: You must specify -m <machine type> or --machine <machine type>"
exit 1
elif [ ! -z "$MACHINE" ] && [ ! -f "${WORKSPACE}/${MACHINE}.coffee" ]; then
echo -e "[ERROR] $0: Invalid machine type specified: $MACHINE"
exit 1
fi
if [ ! -d "$WORKSPACE/layers" ]; then
echo -e "[ERROR] $0: Script must be run from root of yocto project, or specify root via -r <rootdir> or --rootdir <rootdir>"
exit 1
fi
# Get secrets
source $WORKSPACE/../.harmoni-balenaos.env
# Tweak build scripts
sedi () {
sed --version >/dev/null 2>&1 && sed -i -- "$@" || sed -i "" "$@"
}
echo "[INFO] $0: Tweaking build scripts."
# Skip public dockerhub image push
sedi "s/balena_deploy_to_dockerhub/#balena_deploy_to_dockerhub/g" $WORKSPACE/balena-yocto-scripts/automation/jenkins_build.sh
# Use openbalena environment instead of balena-cloud
sedi "s/balena-cloud.com/"$balena_lib_environment"/g" $WORKSPACE/balena-yocto-scripts/automation/include/balena-lib.inc
sedi "s/{_s3_secret_key}/{_s3_secret_key} --endpoint-url https:\/\/s3."$balena_lib_environment"/g" $WORKSPACE/balena-yocto-scripts/automation/include/balena-deploy.inc
# Provide public path to balena-img build image
sedi "s/_namespace=\"\${4}\"/_namespace=\"aggurio\"/g" $WORKSPACE/balena-yocto-scripts/automation/include/balena-deploy.inc
# Change default S3 bucket name
sedi "s/resin-production-img-cloudformation/balena-os/g" $WORKSPACE/balena-yocto-scripts/automation/include/balena-deploy.inc
# Update repo tag
HOSTOS_VERSION="v$(cat $WORKSPACE/VERSION)"
HARMONI_BUILD_TS="$(date +%Y-%m-%dT%H-%M-%S)"
HARMONI_IMAGE_TAG="${HOSTOS_VERSION}.${VARIANT}_${HARMONI_BUILD_TS}"
echo "$HARMONI_IMAGE_TAG" > ${WORKSPACE}/layers/meta-balena-harmoni/recipes-support/harmoni-image-tag/files/HARMONI_IMAGE_TAG
pushd "$WORKSPACE" > /dev/null 2>&1
git tag -a "$HARMONI_IMAGE_TAG" -m "harmoni_jenkins_build"
popd > /dev/null 2>&1
$WORKSPACE/balena-yocto-scripts/automation/jenkins_build.sh -m $MACHINE --shared-dir $WORKSPACE/shared-dir --build-flavor $VARIANT $JENKINS_ARGUMENTS_VAR
# Un-tweak build scripts
echo "[INFO] $0: Un-tweaking build scripts."
sedi "s/#balena_deploy_to_dockerhub/balena_deploy_to_dockerhub/g" $WORKSPACE/balena-yocto-scripts/automation/jenkins_build.sh
sedi "s/"$balena_lib_environment"/balena-cloud.com/g" $WORKSPACE/balena-yocto-scripts/automation/include/balena-lib.inc
sedi "s/{_s3_secret_key} --endpoint-url https:\/\/s3."$balena_lib_environment"/{_s3_secret_key}/g" $WORKSPACE/balena-yocto-scripts/automation/include/balena-deploy.inc
sedi "s/_namespace=\"aggurio\"/_namespace=\"\${4}\"/g" $WORKSPACE/balena-yocto-scripts/automation/include/balena-deploy.inc
sedi "s/balena-os/resin-production-img-cloudformation/g" $WORKSPACE/balena-yocto-scripts/automation/include/balena-deploy.inc
# Clean up build artifacts
rm -rf ${WORKSPACE}/layers/meta-balena-harmoni/recipes-support/harmoni-image-tag/files/HARMONI_IMAGE_TAG
rm -rf $WORKSPACE/deploy-s3
rm -rf $WORKSPACE/deploy-jenkins
You then need to include a bitbake recipe on your device which runs via a cron job or some other regular process to check for updates. If you have your own build of balenaos, Iām assuming you will know how to add a recipe to do that (we called our recipe harmoni-hostapp-update and it runs a cron job every x minutes to check for OS / supervisor updates). Below is the script that is run by the cron job which manages the update process:
#!/bin/bash
# Hostapp update configuration
BALENA_API_ENDPOINT="$(jq --raw-output .apiEndpoint /mnt/boot/config.json)"
BALENA_API_KEY="$(jq --raw-output .deviceApiKey /mnt/boot/config.json)"
DEVICE_TYPE="$(jq --raw-output .deviceType /mnt/boot/config.json)"
FLEET_ID="$(jq --raw-output .applicationId /mnt/boot/config.json)"
DEVICE_UUID="$(jq --raw-output .uuid /mnt/boot/config.json)"
FLEET_TAG_KEY="os_version"
DEVICE_TAG_KEY="os_version"
readarray -t -d "." BALENA_API_ENDPOINT_ARR <<<"$BALENA_API_ENDPOINT"
unset BALENA_API_ENDPOINT_ARR[0]
BALENA_HOST=$(IFS=. ; echo "${BALENA_API_ENDPOINT_ARR[*]}")
TARGET_IMAGE_DIR=/mnt/data/harmoni-hostapp-update
TARGET_IMAGE_FILE="$TARGET_IMAGE_DIR/image.tar.gz"
TARGET_IMAGE_TEMP_DIR="$TARGET_IMAGE_DIR/tmp"
# Get and parse current os image tag
CURR_IMAGE="$(cat /mnt/boot/HARMONI_IMAGE_TAG)"
readarray -t -d "_" CURR_IMAGE_SPLIT <<<"$CURR_IMAGE"
readarray -t -d "." CURR_HOSTOS_VAR <<<"${CURR_IMAGE_SPLIT[0]}"
CURR_VAR=${CURR_HOSTOS_VAR[-1]}
echo "Current hostos version: ${CURR_IMAGE}"
# Determine target os image for fleet and device
FLEET_TARGET_IMAGE=$(curl -s -H "Authorization: Bearer $BALENA_API_KEY" "$BALENA_API_ENDPOINT/v6/application_tag?\$filter=application%20eq%20${FLEET_ID}%20and%20tag_key%20eq%20%27${FLEET_TAG_KEY}%27" | jq -r '.d[0].value')
DEVICE_TARGET_IMAGE=$(curl -s -H "Authorization: Bearer $BALENA_API_KEY" "$BALENA_API_ENDPOINT/v6/device_tag?\$filter=device/uuid%20eq%20%27${DEVICE_UUID}%27%20and%20tag_key%20eq%20%27${DEVICE_TAG_KEY}%27" | jq -r '.d[0].value')
# Set target os image - prioritize device, then fleet, then default to latest
if [ ! -z $DEVICE_TARGET ]; then
TARGET_IMAGE=$DEVICE_TARGET
elif [ ! -z $FLEET_TARGET ]; then
TARGET_IMAGE=$FLEET_TARGET
else
TARGET_IMAGE="latest"
fi
# If target os image is set to latest, determine the latest version
if [ "$TARGET_IMAGE" == "latest" ]; then
# Get all release tags for target device type, sorted reverse by value
RELEASES=$(curl -s -H "Authorization: Bearer $BALENA_API_KEY" "$BALENA_API_ENDPOINT/v6/release_tag?\$filter=release/any(r:r/belongs_to-application/any(bta:bta/slug%20eq%20%27balena_os/${DEVICE_TYPE}%27))&\$orderby=value%20desc" | jq -r '.d | map( [(.value)] ) | add')
declare -a RELEASES_ARR=($(jq -r '.[]' <<<"$RELEASES"))
for i in "${RELEASES_ARR[@]}"; do
readarray -t -d "_" TARGET_IMAGE_SPLIT <<<"$i"
readarray -t -d "." TARGET_HOSTOS_VAR <<<"${TARGET_IMAGE_SPLIT[0]}"
TEST_VAR=${TARGET_HOSTOS_VAR[-1]}
if [ "$TEST_VAR" == "$CURR_VAR" ]; then
TARGET_IMAGE="$i"
break
fi
done
else
TARGET_IMAGE="$TARGET_IMAGE"
fi
echo "Target hostos version: ${TARGET_IMAGE}"
# Determine if device is running its target image
if [ "$CURR_IMAGE" == "v$TARGET_IMAGE" ]; then
echo "Target hostos already installed!"
else
echo "Target hostos not installed, downloading target..."
# Get image from release (uses first image, should only be one)
RELEASE_ID=$(curl -s -H "Authorization: Bearer $BALENA_API_KEY" "$BALENA_API_ENDPOINT/v6/release_tag?\$filter=release/any(r:r/belongs_to-application/any(bta:bta/slug%20eq%20%27balena_os/${DEVICE_TYPE}%27))%20and%20tag_key%20eq%20%27version%27%20and%20value%20eq%20%27${TARGET_IMAGE}%27" | jq -r '.d[0].release.__id')
IMAGE_ID=$(curl -s -H "Authorization: Bearer $BALENA_API_KEY" "$BALENA_API_ENDPOINT/v6/image-is_part_of-release?\$filter=is_part_of-release%20eq%20${RELEASE_ID}" | jq -r '.d[0].image.__id')
# Get target image and manifest
IMAGE_RECORD=$(curl -s -H "Authorization: Bearer $BALENA_API_KEY" "$BALENA_API_ENDPOINT/v6/image?\$filter=id%20eq%20${IMAGE_ID}")
IMAGE_LOCATION=$(jq -r .d[0].is_stored_at__image_location <<< "$IMAGE_RECORD")
IMAGE_SERVICE=$(echo $IMAGE_LOCATION | cut -d/ -f1)
IMAGE_REPOSITORY="$(echo $IMAGE_LOCATION | cut -d/ -f2)/$(echo $IMAGE_LOCATION | cut -d/ -f3)"
IMAGE_MANIFEST=$(jq -r .d[0].content_hash <<< "$IMAGE_RECORD")
# Get authorization token for registry from api
JWT=$(curl -s -H "Authorization: Bearer $BALENA_API_KEY" "$BALENA_API_ENDPOINT/auth/v1/token?service=$IMAGE_SERVICE&scope=repository:$IMAGE_REPOSITORY:pull" | jq -r .token)
mkdir -p "$TARGET_IMAGE_TEMP_DIR"
# Get manifest from registry
TARGET_IMAGE_MANIFEST=$(curl -s -H "Authorization: Bearer $JWT" "https://registry.$BALENA_HOST/v2/$IMAGE_REPOSITORY/manifests/$IMAGE_MANIFEST")
# Download config from registry
CONFIG_DIGEST=$(echo "$TARGET_IMAGE_MANIFEST" | jq -r ".config.digest")
CONFIG_FILE="config.json"
curl -s -L -H "Authorization: Bearer $JWT" "https://registry.$BALENA_HOST/v2/$IMAGE_REPOSITORY/blobs/$CONFIG_DIGEST" --output "$TARGET_IMAGE_TEMP_DIR/$CONFIG_FILE"
# Download layers from registry
LAYER_IDX=0
declare -a LAYER_FILES
for LAYER_DIGEST in $(echo "$TARGET_IMAGE_MANIFEST" | jq -r ".layers[].digest"); do
LAYER_FILES[$LAYER_IDX]="layer$(expr $LAYER_IDX + 1).tar.gz"
curl -s -L -H "Authorization: Bearer $JWT" "https://registry.$BALENA_HOST/v2/$IMAGE_REPOSITORY/blobs/$LAYER_DIGEST" --output "$TARGET_IMAGE_TEMP_DIR/${LAYER_FILES[${LAYER_IDX}]}"
LAYER_IDX=$(expr $LAYER_IDX + 1)
done
# Build manifest.json for newly downloaded image
MANIFEST_JSON=$(jq --arg config "$CONFIG_FILE" --arg layer "${LAYER_FILES[0]}" '. | .[0].Config=$config' <<< "[]")
for idx in "${!LAYER_FILES[@]}"; do
MANIFEST_JSON=$(jq --arg layer "${LAYER_FILES[${idx}]}" '. | .[0].Layers['"${idx}"']=$layer' <<< "$MANIFEST_JSON")
done
echo "$MANIFEST_JSON" > "$TARGET_IMAGE_TEMP_DIR/manifest.json"
# Create image file and cleanup temp directory
tar -czf "$TARGET_IMAGE_FILE" -C "$TARGET_IMAGE_TEMP_DIR" .
rm -rf "$TARGET_IMAGE_TEMP_DIR"
echo "Updating host OS..."
hostapp-update -f ${TARGET_IMAGE_FILE}
if [ $? -eq 0 ]; then
echo "Host OS update complete."
echo "Cleaning up downloaded items..."
rm -rf ${TARGET_IMAGE_DIR}
echo "Rebooting."
reboot
exit 0
else
echo "Host OS update error."
echo "Cleaning up downloaded items..."
rm -rf ${TARGET_IMAGE_DIR}
exit 1
fi
fi
## Update supervisor
source /etc/balena-supervisor/supervisor.conf
if [ ! -z "$SUPERVISOR_TAG" ]; then
SUPERVISOR_IMAGE=$(curl -X GET --silent -k \
"https://api.balena-cloud.com/v6/supervisor_release?\$top=1&\$select=image_name&\$filter=(supervisor_version%20eq%20%27${SUPERVISOR_TAG}%27)%20and%20(is_for__device_type/any(ifdt:ifdt/is_of__cpu_architecture/any(ioca:ioca/slug%20eq%20%27$(arch)%27)))" \
-H "Content-Type: application/json" | jq -r '.d[].image_name')
SUPERVISOR_VERSION=$(cut -d "/" -f 3 <<< "$SUPERVISOR_IMAGE")
fi
echo "Current supervisor version: ${SUPERVISOR_VERSION}"
SUPERVISOR_TARGET=${SUPERVISOR_VERSION}
if [ -f /mnt/boot/HARMONI_SUPERVISOR_TARGET ]; then
SUPERVISOR_TARGET=$(cat /mnt/boot/HARMONI_SUPERVISOR_TARGET)
SUPERVISOR_TARGET_IMAGE=$(curl -X GET --silent -k \
"https://api.balena-cloud.com/v6/supervisor_release?\$top=1&\$select=image_name&\$filter=(supervisor_version%20eq%20%27${SUPERVISOR_TARGET}%27)%20and%20(is_for__device_type/any(ifdt:ifdt/is_of__cpu_architecture/any(ioca:ioca/slug%20eq%20%27$(arch)%27)))" \
-H "Content-Type: application/json" | jq -r '.d[].image_name')
SUPERVISOR_TARGET_VERSION=$(cut -d "/" -f 3 <<< "$SUPERVISOR_TARGET_IMAGE")
fi
echo "Target supervisor version: ${SUPERVISOR_TARGET_VERSION}"
if [[ "${SUPERVISOR_VERSION}" == "${SUPERVISOR_TARGET_VERSION}" ]]; then
echo "Target supervisor already installed!"
else
echo "Newer supervisor available, updating..."
/usr/bin/update-balena-supervisor -i "${SUPERVISOR_TARGET_IMAGE}"
echo "Supervisor update complete."
fi
exit 0
This script is essentially recreating the OS image that was pushed to the openbalena registry by the build process. The device has access to the registry via its API key. Thereās a lot more detail around all of this stuff which is why I want to ultimately release the custom OS as a project, but I thought this might help anyone who is looking to do this in the meanwhile.
