Symlink in Balena sources causes deployment build failure, works fine with CLI.

My project uses a symlink for the Dockerfile.template which references which of the two Dockerfiles can be used. One builds the application from source, the other installs it from a binary. They are called Dockerfile.build and Dockerfile.binary. Dockerfile.template is a relative symlink to one of these.

using balena push or git push deploys the application correctly, whereas the Balena Deploy method does not and fails in the following manner, it does not see the symlink.

[Info]            Starting build for streamyard, user gh_matthewcroughan
[Info]            Dashboard link: https://dashboard.balena-cloud.com/apps/1683882/devices
[Info]            Building on arm01
[Info]            Pulling previous images for caching purposes...
[Success]         Successfully pulled cache images
[Info]            Starting to build traefik
[Info]            Starting to build octodash
[Info]            Starting to build mdns-publisher
[Info]            Starting to build octoprint
[Success]         Successfully built service mdns-publisher
[Success]         Successfully built service traefik
[Success]         Successfully built service octoprint
[Info]            Uploading images
[Success]         Successfully uploaded images
[Error]           Some services failed to build:
[Error]             Service: octodash
[Error]               Error: the Dockerfile (Dockerfile) cannot be empty
[Info]            Built on arm01
[Error]           Not deploying release.

Hi. What are the arguments you are using for these commands and what is you balena-cli version? And from which directory you are running them?

I tried multiple variations but I cannot reproduce this locally using the latest cli (v12.1.16)

balena-cli works fine, this is not an issue with the balena-cli. I’ve already stated what commands to run, it is simply balena push appname. That works correctly.

I am running

matthew@thinkpad ~ $ balena --version
12.1.11

It is deployment with the balena deploy button that behaves incorrectly.

The issue is discussed by me and the Happy Hour crew a bit here, when we discovered this as a potential issue. Also some playback in the chat from Rich Bayliss who seems to think it might be the symlink causing the issue.

Ah, you mean the deploy button not the deploy command. Let me see what I can find

Ah! I did not even realise deploy was also a feature name in the cli. I have some reading to do I think :sweat_smile:

I can’t get balena push working with your repo from a fresh clone (master branch). Neither from the root dir nor from the octodash dir (with --noparent-check).

[main]     dpkg-deb: error: 'octodash.deb' is not a Debian format archive
[main]     
[main]     dpkg: error processing archive octodash.deb (--install):
[main]      dpkg-deb --control subprocess returned error exit status 2
[main]     
[main]     Errors were encountered while processing:
[main]      octodash.deb
[main]     
[main]     The command '/bin/sh -c curl -sL https://github.com/UnchartedBull/OctoDash/releases/download/v""$VERSION""/octodash_""$VERSION""_$(uname -m).deb -o octodash.deb &&     dpkg -i --ignore-depends=libnotify4,xdg-utils,libappindicator3-1,libsecret-1-0 octodash.deb &&     rm octodash.deb' returned a non-zero code: 1

are you doing a balena push from another branch/repo?

From the master branch, using balena push appname I am able to build Octodash fine and that error you presented does not occur for me. Maybe something happened via github in that instant that caused curl to fail.

octodash]        Step 8/9 : RUN curl -sL https://github.com/UnchartedBull/OctoDash/releases/download/v""$VERSION""/octodash_""$VERSION""_$(uname -m).deb -o octodash.deb &&     dpkg -i --ignore-depends=libnotify4,xdg-utils,libappindicator3-1,libsecret-1-0 octodash.deb &&     rm octodash.deb
[octodash]         ---> Running in 06a55d14ce85
[octodash]        Selecting previously unselected package octodash.
[octodash]        (Reading database ... 23580 files and directories currently installed.)
[octodash]        Preparing to unpack octodash.deb ...
[octodash]        Unpacking octodash (1.5.0) ...
[octodash]        Setting up octodash (1.5.0) ...
[octodash]        /var/lib/dpkg/info/octodash.postinst: line 10: update-desktop-database: command not found
[octodash]        
[octodash]        Processing triggers for hicolor-icon-theme (0.17-2) ...
[octodash]        Removing intermediate container 06a55d14ce85
[octodash]         ---> 4a0093012f04
[octodash]        Step 9/9 : CMD ["bash","/usr/src/app/start.sh"]
[octodash]         ---> Running in dde7556f83f3
[octodash]        Removing intermediate container dde7556f83f3
[octodash]         ---> 59efc48c66e3
[octodash]        Successfully built 59efc48c66e3
[Info]            Generating image deltas from release 13661c5454f074494b614598dc707cf0 (id: 1432670)
[=================================>                 ] 67%

Additionally, you can see on the video from the timecode I’ve provided that I am building it correctly via the CLI in the background, and minutes later comment on how it has finished successfully.

Ah x86 isn’t supported by the looks of it? I don’t have ARM hardware with me right now but I’ll take a look at how the button works

Yeah thanks to issues like these, I cannot make a MULTI-ARCH image, nor can I sensibly do this without having to duplicate effort by making multiple Dockerfiles, just to change one link, something I don’t want to get involved with. A feature which upstream Docker has had for a long time (2017?) is doing this multi-arch recognition stuff and having cases in Dockerfiles for each arch.

My Dockerfile.build defines a build procedure, but it fails on the balena build server, probably nothing to do with the server, but perhaps the application itself, being managed by NPM and electron-builder not supporting ARM, but would probably work fine elsewhere. You can see that issue here.

The arch depends on the application itself, and if you have a Dockerfile.template you can use %%BALENA_MACHINE_NAME%% and %%BALENA_ARCH%% placeholders anywhere. For example:

FROM balenalib/%%BALENA_MACHINE_NAME%%-debian:stretch

RUN echo %%BALENA_ARCH%%

Which is a separate thing to the balena-cli issue you linked. See https://www.balena.io/docs/reference/base-images/base-images/. This is usually sufficient to deal with multi-arch issues.

I know these things. It’s just that this is not ideal, since Docker solves this with MULTI-ARCH images using Docker Manifests. Something which Balena does not support in the build server. Your templating scheme does not solve the fundamental problem. Balena is an old version of Docker without these features, which is why this is so problematic when using Docker hub images. The only way to get around this is to base everything in my repo on Balena base images, which is not something I want to do.

This also means issues like this are caused, maybe we should move to this thread as it’s more on topic there. Inconsistencies between LOCAL and REMOTE build procedures

Hi there, the CLI will resolve symlinks and replace them with the real content running balena push, which is why doing balena push works . git push or Deploy with balena button, however, send the symlink to the builder as-is, but Docker/balenaEngine don’t follow symlinks when executing COPY commands (see: https://github.com/moby/moby/issues/6094), so the build fails. Having said that, we would recommend considering not using symlinks, since that will only works using the CLI.