We have a Node.js project and are attaching a remote debugger to an instance of Node running on a device in local mode.
Depending on whether the container should be used for production or debugging, the CMD command in the Dockerfile should be sth like CMD node src/main.js
or CMD node --inspect=0.0.0.0:9229 --nolazy src/main.js
We wanted to use a multi-stage Docker build to achieve this duality of options, like this:
# previous steps...
FROM buildresult as development
ARG DEBUG_PORT=9229
ENV NODE_ENV development
CMD node --inspect-brk=0.0.0.0:${DEBUG_PORT} --nolazy src/main.js
FROM buildresult as production
ENV NODE_ENV production
CMD node src/main.js
Because the production target is the last, this should work fine with the Balena builder. But it seems that we cannot specify a --target option, like it would be possible with docker build, when using the Balena CLI (and specifically balena local push). Is there any way of achieving the desired result?
Hey @maciej.ldc, the right way to do this split in the execution is likely not the multistage build, but by a split in the application start. For example you can add a start script, start.sh, with content similar to this:
# contents of start.sh
#!/usr/bin/bash
if [ -n "$DEVELOPMENT" ]; then
# This will run if the DEVELOPMENT environment variable is set to any value
node --inspect-brk=0.0.0.0:${DEBUG_PORT} --nolazy src/main.js
else
node src/main.js
fi
and then in your CMD set to CMD ["bash", "start.sh](not to forget copying the file in). After this, for devices where you set the DEVELOPMENT env var to a non-empty value it will run the that command line, or if it’s not set, it will run the production command line.
But what if we need to distinguish between development and production images?
For example, what if we want to install ‘vim’ on the development image but not on the production image:
FROM production AS development
RUN install_packages vim
FROM alpine AS production
COPY . .
with Docker, I can do:
docker build --target production -t my_image:production
docker build --target development -t my_image:development
Q1. Am I correct in using the above commands and Dockerfile to distinguish between the two?
Q2. Does balena support this? (The balena-cli docs do not mention --target nor -t)
Yes, as in the example I gave, I want some devices to run a development version of my app. The development version might have additional tools, or might have anything else that might be different than production.
I don’t necessarily want to use local mode for development. Local mode does the balena build on the target. I want to do the balena build on my laptop/desktop.
Regarding balenaCLI not supporting the --target option and other docker build options, I think it’s a mistake. Usually, whenever anyone provides a wrapper around a tool, the wrapper takes its own options, but provides an option to pass the original options to the tool. For example, the gcc wrapper allows me to pass assembler options to the assembler (-Wa,option) and linker options to the linker (-Wl,option). The balenaCLI should have an option to allow me to pass any option I want to the ‘docker build’ command, otherwise, it’s an artificial limitation. Please, could you ask your development team if they would consider this? Is there a place where I can post this suggestion?
Thanks!
If you want different services - you will have to do one of 2 things
have 2 separate apps for these and manage them separately
have some environment variables that you set per device, which enables or disables some services
I know the above options have been discussed above, but I don’t see any other way around it.
About the docker bit, we aren’t exactly a wrapper on top of vanilla docker. We’ve made considerable changes to it to make it ideal for IoT devices. You can find out more about balenaEngine - which is based off Moby from docker here - balenaEngine - A container engine purpose-built for IoT devices . Needless to say, this is completely open source - GitHub - balena-os/balena-engine: Moby-based Container Engine for Embedded, IoT, and Edge uses . One of the main reason for doing this is size - balenaEngine is 3.5x smaller than Docker CE. Things like these are necessary for edge and IoT devices. This of course comes with sacrifices, and looks like you stumbled across one of them
If you feel strongly about it, feel free to take a look at the source, and open an issue on this repo! I’ll also ping the engineer working on this specific part and see if this is something we can enable easily
@anujdeshpande thanks for clarifying that Balena cli is not a wrapper around docker cli. The more I learn about Balena, the more I appreciate the huge effort that you folks have put it. I guess what got me confused is the following sentence: “In order to build containers you will need to have Docker installed on your development machine, and you should be able to execute Docker commands as a non-root user.” from Deploy to your fleet - Balena Documentation
Going back to the example I gave earlier, could you please clarify how I can use env vars to tell Dockerfile.template to generate 2 different images, one for development which contains vim, and one for production which doesn’t contain vim? I’m trying to distinguish between the 2 images at build-time not at runtime. I don’t think the Dockerfile syntax allows for if-then-else style of flow control, but maybe you have a suggestion on how I an achieve this? I also looked at using --buildArg
As for having 2 separate apps, would it work if I have one source directory and (as part of my build system) I generate 2 dockerfiles, e.g. Dockerfile.production and Dockerfile.development? I read the Project Structure but the distinction between Dockerfile.* files seems to only happen based on devicetype and arch. They’re not really 2 apps, since the source code is the same. It’s just that the production image is a subset of the development image.
Can I use the --source flag to only point to the Dockerfile.template? e.g.
and not to the whole source tree?
Is there anything in balena build which could be equivalent to docker build -f ?
or can I use docker-compose.yml to achieve something like this ?
the env vars method is for runtime, not build time. So won’t work for you unfortunately.
if you are using balena push AppName --source, that unfortunately looks for a directory and not a Dockerfile. So it won’t work if you passed that in this case. Have you used balena deploy? This would let you point to a dockerfile with the --dockerfile <dockerfile> flag. My colleague has explained the difference between balena deploy vs balena push - balena push vs build and deploy - #2 by dfunckt
I think this should work well for you
the development and production images are for the underlying balenaOS - and not for the user’s containers. The development image has some things like a passwordless root ssh access. You can find more about it here - What is balenaOS? - Balena Documentation
The idea is that you will use the development image for device that you want to quickly debug during the development cycle. And the production images are what you actually flash on the devices that you deploy. Local mode is one of our main advantages that allows you to rapidly make changes to a development image.
the env vars method is for runtime, not build time. So won’t work for you unfortunately.
if you are using balena push AppName --source, that unfortunately looks for a directory and not a Dockerfile. So it won’t work if you passed that in this case. Have you used balena deploy? This would let you point to a dockerfile with the --dockerfile <dockerfile> flag. My colleague has explained the difference between balena deploy vs balena push - balena push vs build and deploy - #2 by dfunckt
I think this should work well for you
the development and production images are for the underlying balenaOS - and not for the user’s containers. The development image has some things like a passwordless root ssh access. You can find more about it here - What is balenaOS? - Balena Documentation
The idea is that you will use the development image for device that you want to quickly debug during the development cycle. And the production images are what you actually flash on the devices that you deploy. Local mode is one of our main advantages that allows you to rapidly make changes to a development image.