Balena CLI push taking a very long time

Product support
1

Hi there,

Thanks so much for such a great product - we’re 1.5 years into operations after our renewal project, and Balena Cloud has made developing, updating, and deploying our ~400 devices so easy.

We’re updating one of our museum’s digital players, Spotlights, and are running into an issue where the Docker image is taking a very long time (~5 hours) to upload to Balena Cloud when we run the CLI command balena push s__spotlights-x86.

To enable good GPU performance in Chromium we build a base image with VAAPI flags acmilabs/spotlights and use that in our Fleet application Dockerfile.

Question: is there something in our setup that’s causing the base image not to be cached and uploading ~7GB to Balena? The Docker Hub image appears to be ~580MB.

Once that upload finishes, we also build a multi-container fleet to the museum which includes some monitoring containers. When we push to this fleet, the spotlights container is cached, so uploading is very quick.

Any ideas how we can speed things up for the initial single container push?

1 Like
2

Hey @sighmon thanks for the kind words - always great to hear that the products we work so hard on are helping people out in the real world :slight_smile:

Down to business! Where is the 7GB figure coming from? When you perform a balena push with the CLI, everything in that directory is being compressed and uploaded in order for the build to be carried out on the balena builders. What would also be helpful to know is if the upload you’re referring to here that’s taking the time is the one at the start of the build process or at the end when the image is stored. The first is an upload from your machine to the builder, whereas the one at the end is an upload from our builder to the registry. It may also help to share build logs.

Could you also clarify how large the releases are when you look at them on the balenaCloud dashboard? You can find that by going to your Fleet > Releases > Selecting a release, then all the sizes of each of the services are listed. Do these add up to 7GB?

3

@chrisys

Down to business! Where is the 7GB figure coming from?

This is at the upload from my directory stage. You can see the difference in these releases between when we make a code change, and when it’s loading a cached image.

First push after a code change 8.1GB:

fresh-build
fresh-build3660×2606 622 KB

Push without any code change 1.5GB:

cached-push
cached-push3660×2606 607 KB

The directory size I’m pushing:

$ du -sh spotlights 
451M	spotlights

Build log from the successful push (cached):

$ balena push s__spotlights-x86
--------------------------------------------------------------------------------
[Warn] Node.js version "14.19.3" does not satisfy requirement ">=12.8.0 <13.0.0"
[Warn] This may cause unexpected behavior.
--------------------------------------------------------------------------------
[Info]     Starting build for s__spotlights-x86, user g_acmi_developer
[Info]     Dashboard link: https://dashboard.balena-cloud.com/apps/1610632/devices
[Info]     Building on x64_01
[Info]     Pulling previous images for caching purposes...
[Success]  Successfully pulled cache images
[main]     Step 1/14 : FROM acmilabs/spotlights:v2
[main]      ---> 1f7bd825d6f7
[main]     Step 2/14 : COPY . /code/
[main]     Using cache
[main]      ---> 4e0b3bcd7d72
[main]     Step 3/14 : COPY ./conf/20-intel.conf /usr/share/X11/xorg.conf.d/20-intel.conf
[main]     Using cache
[main]      ---> 39047692ae55
[main]     Step 4/14 : RUN mv /usr/bin/lxpolkit /usr/bin/lxpolkit.bak
[main]     Using cache
[main]      ---> f105e570f584
[main]     Step 5/14 : RUN echo "#!/bin/bash" > /etc/X11/xinit/xserverrc   && echo "" >> /etc/X11/xinit/xserverrc   && echo 'exec /usr/bin/X -s 0 dpms -nolisten tcp "$@"' >> /etc/X11/xinit/xserverrc
[main]     Using cache
[main]      ---> 2f3f52c6f0ec
[main]     Step 6/14 : ENV PYTHONUNBUFFERED 1
[main]     Using cache
[main]      ---> d6b0d822c709
[main]     Step 7/14 : ENV UDEV 1
[main]     Using cache
[main]      ---> cdf2ff165a52
[main]     Step 8/14 : ENV DISPLAY :0
[main]     Using cache
[main]      ---> 294f330ba457
[main]     Step 9/14 : ENV DBUS_SYSTEM_BUS_ADDRESS unix:path=/host/run/dbus/system_bus_socket
[main]     Using cache
[main]      ---> f36c47195cc7
[main]     Step 10/14 : COPY ./requirements/base.txt /code/requirements/base.txt
[main]     Using cache
[main]      ---> c8ff1b4b3a03
[main]     Step 11/14 : COPY ./requirements/prod.txt /code/requirements/prod.txt
[main]     Using cache
[main]      ---> d9be7910e48a
[main]     Step 12/14 : RUN pip3 install -Ur /code/requirements/prod.txt
[main]     Using cache
[main]      ---> 85da40958b7a
[main]     Step 13/14 : WORKDIR /code/
[main]     Using cache
[main]      ---> fc422fc2ea4d
[main]     Step 14/14 : CMD ["bash","scripts/x86.sh"]
[main]     Using cache
[main]      ---> 1bb68fd05cf4
[main]     Successfully built 1bb68fd05cf4
[Info]     Uploading images
[Success]  Successfully uploaded images
[Info]     Built on x64_01
[Success]  Release successfully created!
[Info]     Release: 3bb688bb567ade73928d8233d22f0464 (id: 2220510)
[Info]     ┌─────────┬────────────┬────────────┐
[Info]     │ Service │ Image Size │ Build Time │
[Info]     ├─────────┼────────────┼────────────┤
[Info]     │ main    │ 1.36 GB    │ < 1 second │
[Info]     └─────────┴────────────┴────────────┘
[Info]     Build finished in 1 minute, 12 seconds
			    \
			     \
			      \\
			       \\
			        >\/7
			    _.-(6'  \
			   (=___._/` \
			        )  \ |
			       /   / |
			      /    > /
			     j    < _\
			 _.-' :      ``.
			 \ r=._\        `.
			<`\\_  \         .`-.
			 \ r-7  `-. ._  ' .  `\
			  \`,      `-.`7  7)   )
			   \/         \|  \'  / `-._
			              ||    .'
			               \\  (
			                >\  >
			            ,.-' >.'
			           <.'_.''
			             <'

Build logs from a timeout:

$ balena push s__spotlights-x86                        
--------------------------------------------------------------------------------
[Warn] Node.js version "14.19.3" does not satisfy requirement ">=12.8.0 <13.0.0"
[Warn] This may cause unexpected behavior.
--------------------------------------------------------------------------------

   ╭────────────────────────────────────────────────────────────────────╮
   │                                                                    │
   │                  Update available 13.3.0 → 13.6.1                  │
   │                                                                    │
   │   https://github.com/balena-io/balena-cli/blob/master/INSTALL.md   │
   │                                                                    │
   ╰────────────────────────────────────────────────────────────────────╯

EPIPE: write EPIPE

Additional information may be available with the `--debug` flag.

For further help or support, visit:
https://www.balena.io/docs/reference/balena-cli/#support-faq-and-troubleshooting
6

Hello,

Have you tried doing a local build/push to see if it affects the image size significantly? Develop locally - Balena Documentation
Just to test, I pushed acmilabs/spotlights:v2 image locally and it showed 608.5MB size.

12

Unfortunately the caveats of local build don’t work for us - our team works remote, so needs to see shared logs, be able to deploy environment variable changes without code changes, and restart individual containers if necessary.