how to use balena deploy to build on device running development image

I need to build without sending the source code to balenaCloud. In the overview here, it indicates:

Blockquote It is not necessary to install Docker on your development machine if you choose to use a device running balenaOS to build the images (a development image is then required), by specifying a docker daemon’s IP address and port number with the relevant command-line options.

I want to build the image locally on the device running the balenaOS development image and then have that deployed via balenaCloud. However, I am unable figure out what command I need to use to achieve this. All the “balena deploy” command options seem to be for the build being performed on the development machine.

I can perform balena push <IP_address> but that image is lost once I go back to non-local mode.

Can you please advise?

Hi, you can run balena deploy with --dockerHost and --dockerPort options. In this case cli will use docker daemon of the device and upload built image to balenaCloud image registry.

The command will look like this:

balena deploy --build --dockerPort 2375 --dockerHost <host>

Replacing with the device’s IP address (or a .local domain name). Note that the device must be running a development (not production) balenaOS image.

Thanks for the help. I’ve tried the following (replacing app_name appropriately)
balena deploy app_name --build --dockerPort 2375 --dockerHost 192.168.168.77

The result:

    [debug] original argv0="balena" argv=[/home/ge/balena-cli/balena,/snapshot/versioned-source/bin/balena,deploy,app_name,--build,--dockerPort,2375,--dockerHost,192.168.168.77] length=9
    [Debug]   Parsing input...
    Docker seems to be unavailable. Is it installed and running?

I’ve got a proxy environment. I first perform a balena login with BALENARC_PROXY set appropriately and that works. After that, if I unset the proxy, and try the above balena deploy command, I get the error (although I had previously logged in):
You have to log in to continue

If I set the BALENARC_PROXY appropriately, then I get the error:
Docker seems to be unavailable. Is it installed and running?

Since the communication to the device at the local IP 192.168.168.77 appears to be via HTTP but communication to balenaCloud requires the proxy, I’m wondering whether the proxy setting is interfering with communication to the device.

Any thoughts on the right settings to resolve this?
Thanks much!

Hi, I will pass this to our CLI team as I am not sure whether the deploy command reaches our servers or not. Just in case to ask you: you are using a dev image, and not a prod one, right?
Thanks,
Zahari

Yes, I’m using a dev image.
For this experiment, I’m simply using the instructions from here to set up a Raspberry Pi 3.
https://www.balena.io/docs/learn/getting-started/raspberrypi3/nodejs/

The instructions work fine as shown. I’m just trying to learn enough with the Pi so that I can use the methodology to get this working with our own hardware where I can’t push source code outside the corporate environment.

Hi,
The proxy settings should indeed affect communication with a docker daemon on your device because the settings are applied globally for all HTTP requests made by the CLI.
Our team is reviewing what we can do for this case, when HTTP calls need to be made to both a local network device and to the cloud.

Meanwhile, there is a possible workaround if we separate build and deploy commands.
You could use balena build command to build the app using your balenaOS device with

balena build -h <deviceIP> -p <deviceDockerPort> --deviceType raspberrypi3 --arch armv7hf

Note that I’m not specifying the application parameter, but provide device type and target arch info instead to avoid communication with the cloud. Once the build finishes, you can transfer the images from the device to your local machine:

ssh root@<deviceIP> -p 22222 balena-engine save <my_image> | docker load

And now, having the built image locally, you should be able to do

BALENARC_PROXY=<proxy> balena deploy <my_app> <my_image>

Let us know if it helps.

The steps did work! Thanks for the help.

As I was using a docker container to execute the steps, I had to figure out an approach for (pseudo) docker-in-docker first, so it’s somewhat clumsy to do it. I couldn’t figure out how to automate all the steps as I had to determine the image name first that needed to be uploaded. It’s somewhat risky especially for a different user who doesn’t understand the nuances of the steps as they could push the source code to balenaCloud inadvertently if they didn’t specify the image name in the last command.

Any thoughts on whether the Balena CLI could be updated to work in the kind of proxy-based environment I have? I think it’s typical in a lot of companies so I think it would benefit many others. It would also avoid having to install docker, or worse still, use docker-in-docker if one wanted to dockerize the build and deploy steps. Thanks.

I noticed that the balena CLI already seems to use the https_proxy system environment variable if set even when BALENARC_PROXY is not defined. But I tried setting a no_proxy system environment variable. If support for use of a no_proxy system environment variable could be added, that would be great. Example settings would be:
export no_proxy="localhost,127.0.0.1,192.168.168.*"

Hi @ravi1 ,
thanks for your feedback. An engineer is looking into this and will give you feedback of our thoughts about this.

@ravi, I was running some tests and I created the following two CLI issues:

  1. Proxy fails with Node.js versions >= 10.16.0
  2. Allow configurable proxy exceptions

The first issue is a high priority bug that should be fixed soon. The second issue has lower priority, but includes an additional suggested workaround for your case: running an intermediate Squid proxy server on your local network / machine, which could be configured to selectively forward connections to your upstream corporate proxy, while excluding local private IP addresses like 192.168.x.y. You could try setting that up, and share your experience or stumbling blocks with us, so we can build on it.

Thank you for reaching out in the forums!

As a follow-up to this workaround for single container apps, is there a way to do this for a multi-container application?
I tried using the following to send the 3 images in turn for this example:
balena deploy <my_app> <my_image>

But I didn’t see it working. I’m guessing using 3 separate deploy commands isn’t right since it would not be clear how the device would know to treat the 3 containers together as a multi-container app.

Appreciate your advice.
Thanks.

As Paulo said, we already created issues to handle such situation, but in the meantime these are some other workarounds that you can do:

There are other workarounds that are more complicated, so I would suggest trying out any of the two mentioned above.

@ravi, there is now a CLI pull request, pending review, that implements the “no_proxy” feature:

Question: what is your operating system (Windows, macOS, or Linux) and CLI installation type (standalone zip file, executable installer, or npm)? I ask because, even with that PR, not all installation types would support the feature straight away.

Thanks for working on adding the “no_proxy” feature.

I’m using an Ubuntu 18.04 VM running under VirtualBox on a Windows 10 PC.
In case it matters, the VM is set up with a NAT network adapter which is the default.
The CLI is installed in the Ubuntu VM using the standalone zip file method.

The CLI is installed in the Ubuntu VM using the standalone zip file method

Thanks. I’ll look into adding support for proxy exclusion in that installation method first. (The reason why it matters is that no_proxy support depends on the version of Node.js bundled with the CLI installer.)

Do you have an update on this that I can try?
Thanks.

Hi,

You could use balena tunnel {uuid} - 2375:2375 to bring port 2375 from the device, to your localhost. Maybe that will help here?

Do you have an update on this that I can try?

@ravi, in addition to Rich’s suggestion, CLI v11.24.0 and later has added support for the BALENARC_NO_PROXY variable described in the Proxy Support section of the Readme file. However, as the document points out, it is not yet available in every installation type, in particular not yet available as a standalone zip file for Linux. It is available with the npm installation type for all platforms (Linux, Windows, macOS), for as long as you use Node.js version 10.16.0 or later and CLI version 11.24.0 or later. It is also available in the standalonze zip file for Windows (you mentioned your host OS is Windows 10), though switching between Windows and Linux may reveal compatibility issues elsewhere.

Support for the standazone zip file for Linux will come “soon” – I don’t have an ETA at the moment, but I expect it to be a matter of “weeks rather than months”.

Thanks for the suggestion, Rich and Paulo.
I’ll investigate further and try these out.

I did look into using squid as an intermediate proxy, but that seemed rather complex to configure even in a standard way, and wasn’t sure what needed to be done to achieve the special setup needed here.

Also, I didn’t want to go down the Windows CLI approach as it requires Docker running under Windows which, in my case, would prevent VirtualBox from being usable… I didn’t investigate the legacy Docker Toolbox approach.

Hi @ravi yeah that makes sense, thanks for the feedback. The npm installation probably sounds like the easiest route to me, at least until the standalone is available. Let us know if you manage to get it up and working