I’m starting to get more infra ready for scaling out a wider prototype of my project, and the main balena-related issue I need to track down is that my multi-container app (has my app, grafana, prometheus, browser, audio, etc.) does releases that use balena push in the regular way, but it’s pushing an image for every container in my docker-compose.yaml file, even though the other 6 containers virtually never change. Is there some way to version the other containers separately, so I’m only pushing new versions of my main app? Right now, all test devices download multiple gigs of the same images again with every release, which isn’t super-ideal…
1 Like
Hello @codingparadox first of all welcome to the balena community!
At balena we use deltas, so in theory you should download only the deltas of the containers instead of the complete container. Is this not the behaviour that you see?
Could you please share more details of your releases to see how we can help you more?
I searched for deltas, and found this in our build (via github actions) every time balena push runs:
[Info] Generating image deltas from release 2087462d339fdf2899fe35ce35edd122 (id: 3163602)
[Warning] Failed to generate deltas due to an internal error; will be generated on-demand
The docker-compose file has a mix of our custom-built image by the repo, not shown below, and then this stack of common components, which only change if i specifically change the version number:
# Port 9090
prometheus:
network_mode: host
image: ghcr.io/boatkit-io/docker-images/prometheus:v2.52.0
volumes:
- prometheus-data:/prometheus
# Port 3200
grafana:
network_mode: host
image: ghcr.io/boatkit-io/docker-images/grafana:v10.1.10
depends_on:
- "prometheus"
environment:
GF_SERVER_HTTP_PORT: 3200
GF_SERVER_ROOT_URL: "%(protocol)s://%(domain)s:%(http_port)s/grafana/"
GF_SERVER_SERVE_FROM_SUB_PATH: "true"
GF_AUTH_ANONYMOUS_ENABLED: "true"
GF_AUTH_ANONYMOUS_ROLE: Viewer
GF_AUTH_ANONYMOUS_HIDE_VERSION: "true"
GF_SECURITY_ALLOW_EMBEDDING: "true"
volumes:
- grafana-data:/var/lib/grafana
browser:
image: ghcr.io/boatkit-io/docker-images/balena-browser:v2.7.1
network_mode: host
privileged: true
depends_on:
- "goatkit"
environment:
LAUNCH_URL: "http://localhost"
KIOSK: 0
PERSISTENT: 1
ENABLE_GPU: 1
WINDOW_SIZE: 1920,1080
labels:
io.balena.features.dbus: "1"
volumes:
- browser-data:/data
wifi-connect:
image: ghcr.io/boatkit-io/docker-images/wifi-connect:v4.11.45
network_mode: host
privileged: true
cap_add:
- NET_ADMIN
environment:
PORTAL_SSID: "Boatkit Setup"
PORTAL_LISTENING_PORT: 88
DBUS_SYSTEM_BUS_ADDRESS: "unix:path=/host/run/dbus/system_bus_socket"
labels:
io.balena.features.dbus: "1"
# Port 8080
mjpg-streamer:
image: ghcr.io/boatkit-io/docker-images/mjpg-streamer:v0.0.2
network_mode: host
privileged: true
# Port 4317
balenaaudio:
image: bh.cr/balenalabs/audio-rpi
privileged: true
labels:
io.balena.features.dbus: "1"
ports:
- 4317:4317 # Only required if using PA over TCP socket
The end of the balena push shows all the images being pushed:
[Info] Release: 3e5f9bbf0c4af49d5f1d09079e832828 (id: 3163683)
[Info] ┌───────────────┬────────────┬────────────┐
[Info] │ Service │ Image Size │ Build Time │
[Info] ├───────────────┼────────────┼────────────┤
[Info] │ goatkit │ 282.34 MB │ 2 seconds │
[Info] ├───────────────┼────────────┼────────────┤
[Info] │ prometheus │ 253.56 MB │ < 1 second │
[Info] ├───────────────┼────────────┼────────────┤
[Info] │ grafana │ 368.16 MB │ < 1 second │
[Info] ├───────────────┼────────────┼────────────┤
[Info] │ browser │ 1.30 GB │ < 1 second │
[Info] ├───────────────┼────────────┼────────────┤
[Info] │ wifi-connect │ 124.83 MB │ < 1 second │
[Info] ├───────────────┼────────────┼────────────┤
[Info] │ mjpg-streamer │ 155.11 MB │ < 1 second │
[Info] ├───────────────┼────────────┼────────────┤
[Info] │ balenaaudio │ 67.50 MB │ < 1 second │
[Info] └───────────────┴────────────┴────────────┘
… and when the build pushes out, it updates to a new version of all images, which each download one by one, with especially browser taking forever (since it’s a 1.3GB download).
1 Like
Hello @codingparadox
I checked and if i’m not wrong, this is known issue, and this consistently happens when referencing external images in the docker-compose (referenced as image instead of build).
So maybe you can try to build the browser container instead of using an external image to take advantage of the deltas and check if this improves the experience.
If I just have a bunch of local dockerfiles that are just FROM [current image path], and the current image path never changes, will that work? Delta should be 0 every time.
For those searching this later, yes, just making all of our containers single-line Dockerfiles with just FROM ghcr.io/boatkit-io/... triggers all of the nice delta calculation stuff to realize the delta is 0 bytes and it works great. I switched to balena doing the docker build of our main app too (simple 5-line docker build), and so it’s detecting deltas as a few kilobytes as one would hope.
4 Likes
@codingparadox glad to hear that it worked and thanks for sharing the solution!
Let us know if we can help you more!