No logging output in local mode

Hey!

Im developing a python application on a Pi zero on my local network. At the moment the application is just an infinite loop flashing an led-strip.

Building in local mode works as expected and the led-strip does flash as expected, i dont get any logs in the console though. Nothing is shown after [Live] Device state settled.
No luck running balena logs with the .local adress or the ip address of the device either.

I do get the correct logs when pushing to the Balena cloud builder and running in the cloud environment though.

Hi
Can you share the application source code here?

absolutely, made main.py super simple to demonstrate the problem.

main.py

import time

if __name__ == '__main__':
    while True:
        print('This should print to console!')
        time.sleep(1)

Dockerfile

FROM balenalib/raspberry-pi-debian-python:3.5-build

WORKDIR /app

COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

docker-compose.yml

version: '2'

services:

  test_app:
    container_name: test_app
    build:
      dockerfile: Dockerfile
    command: [ "python", "./main.py" ]
    privileged: true

Hi @hasse, thanks for sharing code. I cannot reproduce this issue on my end running your code on my RPi3. I have couple of questions to troubleshoot.

  1. When it’s in local mode: Can you confirm that the device is accessible from the local address? Is that address getting resolved within your network? Could you ping the device address?

  2. What’s the balenaOS version that this device is running?

  3. Could you also share your development setup? What’s the operating system you use? What’s the version of balena-cli and how did you install it?

  4. Could you run the balena-cli commands with DEBUG=1 prefix?
    Please copy paste the results, especially anything that looks out of ordinary.

Cheers…

Hi,

  1. Pinging the device from a browser returns OK instantly. Using ping in bash works great too.

  2. balenaOS 2.46.1+rev1

  3. Im on Ubuntu 18.04.4 LTS. Balena cli version 11.28.2 installed by downloading the binary from the balena-io/balena-cli releases and moving it to /usr/local/bin/

  4. Output of all commands run can be found below:

sudo balena scan
  host:          820b04a.local
  address:       192.168.1.143
  dockerInfo: 
    Containers:        2
    ContainersRunning: 2
    ContainersPaused:  0
    ContainersStopped: 0
    Images:            17
    Driver:            aufs
    SystemTime:        2020-02-24T18:03:27.593044421Z
    KernelVersion:     4.19.71
    OperatingSystem:   balenaOS 2.46.1+rev1
    Architecture:      armv6l
  dockerVersion: 
    Version:    18.09.10-dev
    ApiVersion: 1.39
DEBUG=1 balena push 820b04a.local
[debug] original argv0="balena" argv=[/usr/local/bin/balena,/snapshot/versioned-source/bin/balena,push,820b04a.local] length=4
[debug] Using . as build source
[Debug]   Checking we can access device
[Debug]   Sending request to http://820b04a.local:48484/ping
[Debug]   Checking device version: 10.6.27
[Info]    Starting build on device 820b04a.local
[Debug]   Loading project...
[Debug]   Resolving project...
[Debug]   docker-compose.yml file found at "."
[Debug]   Creating project...
[Debug]   Tarring all non-ignored files...
[Debug]   Sending request to http://820b04a.local:48484/v2/local/device-info
[Debug]   Found build tasks:
[Debug]       test_app: build [gardener]
[Debug]   Resolving services with [raspberry-pi|rpi]
[Debug]   Found project types:
[Debug]       test_app: Standard Dockerfile
[Debug]   Probing remote daemon for cache images
[Debug]   Using 14 on-device images for cache...
[Debug]   Starting builds...
[Build]   [test_app] Step 1/7 : FROM balenalib/raspberry-pi-debian-python:3.5-build
[Build]   [test_app]  ---> f2857e341c99
[Build]   [test_app] Step 2/7 : WORKDIR /app
[Build]   [test_app]  ---> Using cache
[Build]   [test_app]  ---> 437dd30d6234
[Build]   [test_app] Step 3/7 : COPY requirements.txt ./
[Build]   [test_app]  ---> Using cache
[Build]   [test_app]  ---> 8256ec162024
[Build]   [test_app] Step 4/7 : RUN pip install --no-cache-dir -r requirements.txt
[Build]   [test_app]  ---> Using cache
[Build]   [test_app]  ---> ca130d41cdea
[Build]   [test_app] Step 5/7 : COPY . .
[Build]   [test_app]  ---> fbc5e4dfd7d9
[Build]   [test_app] Step 6/7 : LABEL io.resin.local.image=1
[Build]   [test_app]  ---> Running in aed580c8bcaf
[Build]   [test_app] Removing intermediate container aed580c8bcaf
[Build]   [test_app]  ---> 2205eaaca171
[Build]   [test_app] Step 7/7 : LABEL io.resin.local.service=test_app
[Build]   [test_app]  ---> Running in f2cd9819cdb3
[Build]   [test_app] Removing intermediate container f2cd9819cdb3
[Build]   [test_app]  ---> 1a42c213c0e9
[Build]   [test_app] Successfully built 1a42c213c0e9
[Build]   [test_app] Successfully tagged local_image_test_app:latest
[Debug]   Setting device state...
[Debug]   Sending request to http://820b04a.local:48484/v2/local/target-state
[Debug]   Sending target state: {"local":{"name":"summer-fog","config":{"SUPERVISOR_DELTA_VERSION":"3","HOST_CONFIG_disable_splash":"1","HOST_CONFIG_dtparam":"\"i2c_arm=on\",\"spi=on\",\"audio=on\"","HOST_CONFIG_enable_uart":"1","HOST_CONFIG_gpu_mem":"16","SUPERVISOR_LOCAL_MODE":"1","SUPERVISOR_POLL_INTERVAL":"900000","SUPERVISOR_VPN_CONTROL":"true","SUPERVISOR_INSTANT_UPDATE_TRIGGER":"true","SUPERVISOR_CONNECTIVITY_CHECK":"true","SUPERVISOR_LOG_CONTROL":"true","SUPERVISOR_DELTA":"false","SUPERVISOR_DELTA_REQUEST_TIMEOUT":"30000","SUPERVISOR_DELTA_APPLY_TIMEOUT":"0","SUPERVISOR_DELTA_RETRY_COUNT":"30","SUPERVISOR_DELTA_RETRY_INTERVAL":"10000","SUPERVISOR_OVERRIDE_LOCK":"false","SUPERVISOR_PERSISTENT_LOGGING":"false"},"apps":{"1":{"name":"localapp","commit":"localrelease","releaseId":"1","services":{"1":{"environment":{},"labels":{},"container_name":"test_app","command":["python","./main.py"],"privileged":true,"imageId":1,"serviceName":"test_app","serviceId":1,"image":"local_image_test_app:latest","running":true}},"volumes":{},"networks":{}}}},"dependent":{"config":{}}}
[Debug]   Sending request to http://820b04a.local:48484/v2/local/target-state

[Debug]   Sending request to http://820b04a.local:48484/v2/local/device-info
[Live]    Waiting for device state to settle...
[Debug]   Device state not settled, retrying in 1000ms
[Debug]   Device state not settled, retrying in 1000ms
[Debug]   Device state not settled, retrying in 1000ms
[Debug]   Device state not settled, retrying in 1000ms
[Debug]   Device state not settled, retrying in 1000ms
[Debug]   Device state not settled, retrying in 1000ms
[Debug]   Device state not settled, retrying in 1000ms
[Debug]   Device state not settled, retrying in 1000ms
[Debug]   Device state not settled, retrying in 1000ms
[Debug]   Device state not settled, retrying in 1000ms
[Live]    Device state settled
DEBUG=1 balena logs 820b04a.local
[debug] original argv0="balena" argv=[/usr/local/bin/balena,/snapshot/versioned-source/bin/balena,logs,820b04a.local] length=4
[Debug]   Checking we can access device
[Debug]   Sending request to http://820b04a.local:48484/ping

The only one that looks suspicious is balena logs getting stuck pinging. Tested with both the .local address and the ip of the device. Network issues feel unlikely considering that the local push runs fine.

Hi @hasse,
Thanks for the additional details on this.
I’ve reached the respective internal team to investigate this issue.
I’ve also opened a GitHub issue on our balena-cli repository, which you can use to track progress on this:

Kind regards,
Thodoris

1 Like

I am observing the same issue. I can SSH to the device using the IP address and balena scan identifies the device successfully. The device is running balenaOS 2.47.0+rev1. I’m using a Mac on 10.15.3 and version 11.28.2 of the CLI.

Hi @Archie

Thank you for your message. I’ve added a note with your system configuration to our existing balena-cli issue on Github for this matter ( https://github.com/balena-io/balena-cli/issues/1631 ), which you can use to track progress on this.

Kind regards
Alida

@Archie, what is the device type you were using? (E.g. Raspberry Pi 4, Intel NUC, etc.). I note that @hasse was using a Raspberry Pi Zero. I gather that at least 2 balena engineers have so far been unable to reproduce the issue. I have myself tested with:

  • CLI v11.28.2 (standalone zip install), Ubuntu 18.04, balenaOS 2.46.1+rev1, RPi 3
  • CLI v11.28.2 (standalone zip install), Ubuntu 18.04, balenaOS 2.47.0+rev1, RPi 4
  • CLI v11.28.2 (standalone zip install), macOS 10.14.6, balenaOS 2.46.1+rev1, RPi 3
  • CLI v11.28.2 (standalone zip install), macOS 10.14.6, balenaOS 2.47.0+rev1, RPi 4
  • CLI v11.28.11 (npm install), macOS 10.14.6, balenaOS 2.46.1+rev1, RPi 3
  • CLI v11.28.11 (npm install), macOS 10.14.6, balenaOS 2.47.0+rev1, RPi 4

@hasse, @Archie, is the issue consistently reproducible at your end? (As opposed to being intermittent, or happening at random.) If both of you were using an RPi Zero, then we should specifically try to reproduce it on an RPi Zero. Otherwise, what I am thinking is to produce a new CLI release that prints additional debugging information, so we can narrow it down further. Thank you for reporting it and for helping us debug it!

I’m using a Pi 4. I’ve just tested again at work and it seems to be working OK. Previously I was testing at home. balena scan doesn’t produce any results at work so I am using the IP address. I’ll test later when I’m home to see if the issue is reproducible there and if there is any difference between using the IP address and the uuid.local.

Exactly the same issue on a freshly installed Pi2 as with the Zero.
Stuck at pinging every time i run it, not intermittent or random.

Installing via npm install balena-cli and running npx balena logs {{pi zero}}.local does work though!
I tried NPM installation at the very beginning but it failed because i totally forgot about npx.

This issue could very likely be limited to the standalone CLI imo.

What version did you use for the standalone installation? As we cannot easily reproduce it, any additional info would help in finding the root cause of this and fixing it sooner.

I previously updated the CLI but didn’t test it afterwards. I’m now on 11.28.8 and have just tested and it’s working fine.

Thanks for the additional info, I’ll jot that down in the issue for reference.

I install the cli by:

  1. Downloading the CLI from the repo, https://github.com/balena-io/balena-cli/releases/tag/v11.28.11
  2. unzip
  3. run sudo mv balena /usr/local/bin/balena

I also get this issue in the standalone CLI when trying to login via web auth (works fine in NPM version though):

DEBUG=1 balena login
[debug] original argv0="balena" argv=[/usr/local/bin/balena,/snapshot/versioned-source/bin/balena,login] length=3
 _            _
| |__   __ _ | |  ____  _ __    __ _
| '_ \ / _` || | / __ \| '_ \  / _` |
| |_) | (_) || ||  ___/| | | || (_) |
|_.__/ \__,_||_| \____/|_| |_| \__,_|


Logging in to balena-cloud.com
? How would you like to login? Web authorization (recommended)
Connecting to the web dashboard
null

I tried installing the same version using the installer on MacOS and there were no issues. what OS are you using?

@hasse, thanks for the additional information. The null error is most likely a different known issue: https://github.com/balena-io/balena-cli/issues/1333
A workaround for the null error is described there for some installers and operating systems, but unfortunately not the standalone (zip) installer. Some common causes are also listed in the issue.

As for the balena logs issue, I’ve just been able to reproduce it following your installation steps - thanks for sharing. The problem is that the standalone zip binary cannot be simply copied on its own to a folder like /usr/local/bin/, because it depends on several other files being present on the same folder as the executable:

Required standalone files
$ unzip balena-cli-v11.28.2-linux-x64-standalone.zip
...
$ find balena-cli -type f
balena-cli/balena
balena-cli/ext2fs/build/Release/bindings.node
balena-cli/drivelist/build/Release/obj.target/drivelist.node
balena-cli/drivelist/build/Release/drivelist.node
balena-cli/lzma-native/deps/bin_x86-64/liblzma.dll
balena-cli/lzma-native/deps/bin_i686/liblzma.dll
balena-cli/lzma-native/binding-v4.0.6-node-v64-linux-x64/lzma_native.node
balena-cli/xxhash/build/Release/hash.node
balena-cli/xxhash/build/Release/obj.target/hash.node
balena-cli/etcher-sdk/node_modules/drivelist/build/Release/obj.target/drivelist.node
balena-cli/etcher-sdk/node_modules/drivelist/build/Release/drivelist.node
balena-cli/net-keepalive/node_modules/ffi-napi/build/Release/ffi_bindings.node
balena-cli/net-keepalive/node_modules/ffi-napi/build/Release/obj.target/ffi_bindings.node
balena-cli/net-keepalive/node_modules/ffi-napi/build/Release/node-addon-api/src/nothing.node
balena-cli/net-keepalive/node_modules/ffi-napi/build/Release/nothing.node
balena-cli/net-keepalive/node_modules/ref-napi/build/Release/obj.target/node_modules/node-addon-api/src/nothing.node
balena-cli/net-keepalive/node_modules/ref-napi/build/Release/obj.target/binding.node
balena-cli/net-keepalive/node_modules/ref-napi/build/Release/binding.node
balena-cli/net-keepalive/node_modules/ref-napi/build/Release/nothing.node
balena-cli/etcher-image-write/node_modules/drivelist/build/Release/obj.target/drivelist.node
balena-cli/etcher-image-write/node_modules/drivelist/build/Release/drivelist.node
balena-cli/@balena.io/usb/build/Release/usb_bindings.node
balena-cli/@balena.io/usb/build/Release/obj.target/usb_bindings.node
balena-cli/@balena.io/usb/src/binding/usb_bindings.node
balena-cli/xdg-open
balena-cli/resin-cli-visuals/node_modules/drivelist/build/Release/obj.target/drivelist.node
balena-cli/resin-cli-visuals/node_modules/drivelist/build/Release/drivelist.node
balena-cli/mountutils/build/Release/obj.target/MountUtils.node
balena-cli/mountutils/build/Release/MountUtils.node

Indeed, the “standalone” name is a bit misleading in this regard, because it cannot stand that much alone, it needs some 30 friends with it. :slight_smile: (The non-standalone installation includes more than 55,000 files, mostly javascript text files, whereas the “standalone” installation includes only some 30 files, all compiled binary files.)

I have however tested that a soft link appears to work. So, for example, the installation steps with the “standalone” zip file could be:

$ mkdir -p /opt  # /opt is just an example, could be any other folder
$ cd /opt
$ curl -LO https://github.com/balena-io/balena-cli/releases/download/v11.28.2/balena-cli-v11.28.2-linux-x64-standalone.zip
$ unzip balena-cli-v11.28.2-linux-x64-standalone.zip
$ rm /usr/local/bin/balena
$ ln -s /opt/balena-cli/balena /usr/local/bin/balena
$ ls -l /usr/local/bin/balena
lrwxrwxrwx 1 root root 22 Mar  3 14:48 /usr/local/bin/balena -> /opt/balena-cli/balena

Alternatively, to avoid the soft link, you could add the unzipped balena-cli folder (wherever it is) to the PATH environment variable. The important point is that the balena executable cannot be moved away from the unzipped folder on its own.

Thats it!
I must have assumed that the .zip was some kind of combined release with Etcher because of the folder naming.
Crazy that everything else that i have used Balena-CLI for worked perfectly with just the binary.

May be worth nothing in the NPM installation README that npx has to be used with the latest NPM releases?

Thanks for confirming that it worked, @hasse.

May be worth nothing in the NPM installation README that npx has to be used with the latest NPM releases?

npx should not be needed; the documented command line for the CLI’s NPM installation is:

npm install balena-cli --global --production --unsafe-perm

I’ve just tested it with Node 12.16.1 and npm 6.13.4 to double check. The --global option above installs the CLI in the global Node/npm folders, so the balena command gets automatically added to the PATH. Well, at least depending on how the global Node/npm installation works. I recommend using nvm to install arbitrary versions of Node in your homedir without interfering with the host OS, and not requiring root/sudo permissions for the installation itself. For example, in my case the folder ~/.nvm/versions/node/v12.16.1/bin is in the PATH, and the balena executable (shell script) gets added to that folder when using npm install --global. The INSTALL.md instructions also include a link to a sample Dockerfile that shows the CLI installation steps on an Ubuntu 18.04 Docker image, using nvm.

Hello, I just wanted to let you know that the issue with the null error from the CLI has been closed and fixed in the most recent version.
Thanks again for reporting this.