balena deploy with --source attribute

RodrigoM · April 19, 2022, 8:46pm

Hi there,

We have two architectures to build and want to use different docker-compose.yaml files for balena deploy.
It seems balena deploy does not work nicely with a setup using the --source attribute, which according to documentation allows us to have different docker-compose.yaml files.

We have two folders for each docker-compose.yaml files:
solution-raspi3/docker-compose.yaml
solution-amd64/docker-compose.yaml

balena deploy solution-raspi3 \
--source solution-raspi3 \
--build --projectName solution-aarch64 \
--multi-dockerignore

No matter what we set in the docker-compose.yaml file, the build fails as Docker cannot find the Dockerfiles in the build: of each docker-compose.yaml.

Each docker-compose.yaml has a set a services, each one being something like:

...
  myService:
    build: ./services/myService

When we use a single docker-compose.yaml, it works fine.

TJvV · April 20, 2022, 7:11am

Hi,

The source target is supposed to be a build directory.
From balena deploy --help:

-s, --source <source>                      specify an alternate source directory; default is the working directory

Does it work when you move your specific yaml files to their own directories?
The rest of the files will be resolved relative to the specified source directory.

Edit: sorry, misread your post, missed the part where they were already in separate folders.

Edit 2: just tried out a simple test which seems to work:
Files:

thijs@WSL2:/tmp/balena_src$ cat b1/docker-compose.yml
version: '2'

services:
    b1_main:
        image: balenablocks/dbus
        privileged: true

thijs@WSL2:/tmp/balena_src$ cat b2/docker-compose.yaml
version: '2'

services:
    b2_main:
        image: balenablocks/dbus
        privileged: true

Now when I run the commands:

thijs@WSL2:/tmp/balena_src$ balena deploy Sandbox --build --source b1
[Build]   Building services...
[Build]   b1_main Preparing...
[Info]    Building for rpi/raspberry-pi
[Build]   b1_main latest: Pulling from balenablocks/dbus
^C

thijs@WSL2:/tmp/balena_src$ balena deploy Sandbox --build --source b2
[Build]   Building services...
[Build]   b2_main Preparing...
[Info]    Building for rpi/raspberry-pi
[Build]   b2_main latest: Pulling from balenablocks/dbus

RodrigoM · April 20, 2022, 2:09pm

Thanks @TJvV but it seems you are using an image in your docker-compose.yaml files.
We are using build: to build our own images and that’s when we see the problem.

TJvV · April 20, 2022, 3:56pm

Hi,

When I change my setup to build my own image, it still works.
Can you maybe share your setup, so we can see what might be going wrong?

I’ve changed my setup as follows (b2 is similar and also works):

thijs@WSL2:/tmp/balena_src$ cat b1/docker-compose.yml
version: '2'

services:
    b1_main:
        build: ./b1_build
        privileged: true

thijs@WSL2:/tmp/balena_src$ cat b1/b1_build/Dockerfile
FROM balenalib/raspberrypi3
CMD ["balena-idle"]

Running the same command as before gives:

thijs@WSL2:/tmp/balena_src$ balena deploy Sandbox --build --source b1
[Build]   Building services...
[Build]   b1_main Preparing...
[Info]    Building for rpi/raspberry-pi
[Build]   b1_main Step 1/2 : FROM balenalib/raspberrypi3
[Build]   b1_main  ---> 4c43a961009f
[Build]   b1_main Step 2/2 : CMD ["balena-idle"]
[Build]   b1_main  ---> [Warning] The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
[Build]   b1_main  ---> Running in 1c9b7df02d02
[Build]   b1_main Removing intermediate container 1c9b7df02d02
[Build]   b1_main  ---> 23686efed1ed
[Build]   b1_main Successfully built 23686efed1ed
[Build]   b1_main Successfully tagged b1_b1_main:latest
[Build]   b1_main Image size: 157.40 MB
[Build]   Built 1 service in 0:28

RodrigoM · April 20, 2022, 8:51pm

It does not work for me (using Balena CLI 13.4.0 on macOS).
I have the following structure:

.
├── deploy-aarch64
│   └── docker-compose.yaml
├── deploy-amd64
│   └── docker-compose.yaml
├── services
│   ├── service1
│   │   ├── Dockerfile.aarch64
│   │   ├── Dockerfile.amd64
│   ├── service2
│   │   ├── Dockerfile.aarch64
│   │   ├── Dockerfile.amd64

balena deploy my-app \
--build \
--source deploy-aarch64  \
--projectName project-aarch64 \
--multi-dockerignore \
--release-tag Version "3.0.0" \
--tag 3.0.0

My docker-compose.yaml file looks like this:

version: '2.1'
services:
  service1:
    build: ./services/service1
...
  service2:
    build: ./services/service2

The following error is thrown:

[Build]   service1 Preparing...
[Build]   service2 Preparing...
[Info]    Building for aarch64/raspberrypi3-64
[Info]    Docker Desktop detected (daemon architecture: "x86_64")
[Info]      Docker itself will determine and enable architecture emulation if required,
[Info]      without balena-cli intervention and regardless of the --emulated option.
[Debug]   Found build tasks:
[Debug]       service1: build [./services/service1]
[Debug]       service2: build [./services/service2]
[Debug]   Resolving services with [raspberrypi3-64|aarch64]
[Build]   Built 2 services in 0 seconds
[Error]   Deploy failed
Could not find a Dockerfile for this service

Error: Could not find a Dockerfile for this service
    at /usr/local/lib/balena-cli/node_modules/resin-bundle-resolve/build/index.js:95:32
    at Generator.next (<anonymous>)
    at /usr/local/lib/balena-cli/node_modules/resin-bundle-resolve/build/index.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/usr/local/lib/balena-cli/node_modules/resin-bundle-resolve/build/index.js:4:12)
    at resolveTarStreamOnFinish (/usr/local/lib/balena-cli/node_modules/resin-bundle-resolve/build/index.js:89:12)
    at Object.<anonymous> (/usr/local/lib/balena-cli/node_modules/resin-bundle-resolve/build/index.js:49:19)
    at Generator.next (<anonymous>)
    at /usr/local/lib/balena-cli/node_modules/resin-bundle-resolve/build/index.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/usr/local/lib/balena-cli/node_modules/resin-bundle-resolve/build/index.js:4:12)
    at Extract.<anonymous> (/usr/local/lib/balena-cli/node_modules/resin-bundle-resolve/build/index.js:47:34)
    at Object.onceWrapper (events.js:420:28)
    at Extract.emit (events.js:326:22)
    at Extract.EventEmitter.emit (domain.js:483:12)
    at finishMaybe (/usr/local/lib/balena-cli/node_modules/tar-stream/node_modules/readable-stream/lib/_stream_writable.js:624:14)
From previous event:
    at cloneTarStream (/usr/local/lib/balena-cli/node_modules/tar-utils/build/index.js:72:12)
    at resolveTasks (/usr/local/lib/balena-cli/build/utils/compose_ts.js:575:43)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async performResolution (/usr/local/lib/balena-cli/build/utils/compose_ts.js:564:5)
    at async makeBuildTasks (/usr/local/lib/balena-cli/build/utils/compose_ts.js:541:5)
    at async $buildProject (/usr/local/lib/balena-cli/build/utils/compose_ts.js:141:19)
    at async awaitInterruptibleTask (/usr/local/lib/balena-cli/build/utils/helpers.js:277:16)
    at async buildProject (/usr/local/lib/balena-cli/build/utils/compose_ts.js:127:47)
    at async DeployCmd.deployProject (/usr/local/lib/balena-cli/build/commands/deploy.js:96:37)
    at async DeployCmd.run (/usr/local/lib/balena-cli/build/commands/deploy.js:50:25)
    at async DeployCmd._run (/usr/local/lib/balena-cli/node_modules/@oclif/command/lib/command.js:43:20)
    at async Config.runCommand (/usr/local/lib/balena-cli/node_modules/@oclif/config/lib/config.js:175:24)
    at async CustomMain.run (/usr/local/lib/balena-cli/node_modules/@oclif/command/lib/main.js:27:9)
    at async CustomMain._run (/usr/local/lib/balena-cli/node_modules/@oclif/command/lib/command.js:43:20)
    at async /usr/local/lib/balena-cli/build/app.js:76:13
    at async Promise.all (index 2)
    at async oclifRun (/usr/local/lib/balena-cli/build/app.js:94:5)
    at async Object.run (/usr/local/lib/balena-cli/build/app.js:107:9)
    at async run (/usr/local/lib/balena-cli/bin/run:20:2)```

TJvV · April 21, 2022, 7:44am

Hi,

From the looks of it, your main problem here is what I mentioned earlier.
The directory being searched for a Dockerfile is relative to your source directory.

By specifying --source deploy-aarch64 and build: ./services/service1, you are telling balena to find your Dockerfile in ./deploy-aarch64/services/service1.

If you move your services into the deploy directories, it will work.

From my testing here, this seems to work as long as the services are actually subdirectories of where your docker-compose.yaml is located.

So, an entry
build: ./services/b1
does work, but
build: ../services/b1
does not.

For my tests, I have used a device-specific Dockerfile:

thijs@WSL2:/tmp/balena_src$ ls b1/services/b1/
Dockerfile.raspberry-pi

RodrigoM · April 21, 2022, 2:42pm

Hi,

Okay, but that removes the whole usefulness of the --source attribute. The idea is to have different docker-compose.yaml files for each architecture (e.g. some architectures have different services).
If we have to move all the services into EACH “deploy” folder, we would have to replicate the services inside each “deploy” folder or use symlinks.

Much better off would be to add a parameter --file in balena deploy to point to different docker-compose.yaml instead of the default.

Any suggestions?

RodrigoM · April 21, 2022, 3:02pm

FYI, we decided to use symlinks inside the “deploy” folders. Not super sleek, but it works. Thanks.

TJvV · April 22, 2022, 1:18pm

I think the main use of the --source flag is meant to be able to run the command wherever you want, instead of having to be in the actual directory.

I am currently not aware of a better way to share services between compose files with balena other than building the images for services separately and using those images in your compose files.

For reference, with docker-compose it is possible by doing something like docker-compose --project-directory /tmp/balena_src --file /tmp/balena_src/b1/docker-compose.yml build.

docker-compose:

thijs@WSL2:/tmp/balena_src$ cat b1/docker-compose.yml
version: '2'

services:
  s1:
    build: ./services/s1

  s2:
    build: ./services/s2

services:

thijs@WSL2:/tmp/balena_src$ cat services/s1/Dockerfile
FROM balenalib/raspberrypi3
CMD ["balena-idle"]

Building gives something like this:

thijs@MSI-WSL2:/tmp$ docker-compose --project-directory /tmp/balena_src --file /tmp/balena_src/b1/docker-compose.yml build
Building s1
Step 1/2 : FROM balenalib/raspberrypi3
 ---> 4c43a961009f
Step 2/2 : CMD ["balena-idle"]
 ---> [Warning] The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
 ---> Running in 82af725d89f8
Removing intermediate container 82af725d89f8
 ---> 67c74ffc3a37
Successfully built 67c74ffc3a37
Successfully tagged balena_src_s1:latest
Building s2
Step 1/2 : FROM balenalib/raspberrypi3
 ---> 4c43a961009f
Step 2/2 : CMD ["balena-idle"]
 ---> [Warning] The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/amd64) and no specific platform was requested
 ---> Running in db53da566c76
Removing intermediate container db53da566c76
 ---> 00b3917e53ad
Successfully built 00b3917e53ad
Successfully tagged balena_src_s2:latest

Do note I specified an absolute path for the docker-compose file here.

I think wrapping this functionality of docker-compose or similar could make for a pretty nice feature request.