Localmode push hangs

RSP · August 24, 2020, 10:51am

Hi,

I’m using localmode to push my application to a raspberry pi 4.

The application is currently very simple, just a single container in a docker compose file:

version: '2'
services:

  influxdb:
    build: influxdb
    restart: on-failure
    network_mode: host
    volumes:
      - 'influxdb-data:/var/lib/influxdb'
    environment:
      - INFLUXDB_ADMIN_USER=...
      - INFLUXDB_ADMIN_PASSWORD=...
      - INFLUXDB_DB=...
      - INFLUXDB_USER=...
      - INFLUXDB_USER_PASSWORD=...

volumes:
  influxdb-data:

With influxdb/dockerfile:

FROM arm32v7/influxdb

# Add custom config.
COPY influxdb.conf /etc/influxdb/influxdb.conf

Using “balena push 192.168.20.121” works great the first time, the database is up and running and a I can query and write to it.

But if I ctrl-c the push process and restart it, it hangs after the following:

[Debug]   Sending request to http://192.168.20.121:48484/v2/local/device-info
[Debug]   Found build tasks:
[Debug]       influxdb: build [influxdb]
[Debug]   Resolving services with [raspberrypi4-64|aarch64]
[Debug]   Found project types:
[Debug]       influxdb: Standard Dockerfile
[Debug]   Probing remote daemon for cache images
[Debug]   Using 7 on-device images for cache...
[Debug]   Starting builds...
[Build]   [influxdb] Step 1/4 : FROM arm32v7/influxdb
[Build]   [influxdb]  ---> 1ca48fe485f8
[Build]   [influxdb] Step 2/4 : COPY influxdb.conf /etc/influxdb/influxdb.conf
[Build]   [influxdb]  ---> Using cache
[Build]   [influxdb]  ---> ab245b41c115
[Build]   [influxdb] Step 3/4 : LABEL io.resin.local.image=1
[Build]   [influxdb]  ---> Using cache
[Build]   [influxdb]  ---> 68ec72e9f753
[Build]   [influxdb] Step 4/4 : LABEL io.resin.local.service=influxdb
[Build]   [influxdb]  ---> Using cache
[Build]   [influxdb]  ---> dd4bef6ae22a
[Build]   [influxdb] Successfully built dd4bef6ae22a
[Build]   [influxdb] Successfully tagged local_image_influxdb:latest
[Debug]   Setting device state...
[Debug]   Sending request to http://192.168.20.121:48484/v2/local/target-state

The container is never updated or restarted. Manually restarting the container over SSH does not help either. At this point I can only restart the raspberry pi or disable/enable localmode and run the push command again, until the next time I make changes to the docker-compose.yml file or any of the dockerfiles.

I’m running the latest version of balena-cli, 12.14.11

I’ve never had this problem in any of other balena applications.

xginn8 · August 24, 2020, 1:26pm

Hi @Rune,

What versions of balenaOS and the balena supervisor are you running on your device?

RSP · August 24, 2020, 1:39pm

balenaOS 2.53.12+rev1
supervisor: 11.9.9

My other applications are running:

balenaOS 2.51.1+rev1
supervisor: 11.4.10

I’ll try balenaOS 2.51.1+rev1 and see if it helps

gustavopch · August 24, 2020, 1:59pm

I have exactly the same problem. Posted about it yesterday: Stuck on "Sending request to http://10.0.0.8:48484/v2/local/target-state" with balena push in local mode

RSP · August 24, 2020, 2:28pm

@gustavopch Guess I didnt search the forums thourougly enough it seems

Anyway, using balenaOS 2.51.1+rev1 with supervisor 11.4.10 my problems are gone

xginn8 · August 24, 2020, 3:01pm

The bug that is preventing your localmode push from succeeding was fixed in v11.11.4 of the supervisor (changelog is here, if curious: https://github.com/balena-io/balena-supervisor/blob/master/CHANGELOG.md#v11114). We are working on getting an OS release out for all devices with that fix, so stay tuned for a new host OS.

Additionally, we are working on easier ways to update the supervisor independently from the host OS, which we hope to release in the near term. Thanks for reporting this issue!

xginn8 · January 25, 2021, 4:49pm

Hi there,

I wanted to reach out and let you know you can now upgrade a device’s supervisor directly using the balenaCloud dashboard – those docs are here: https://www.balena.io/docs/reference/supervisor/supervisor-upgrades/

Thanks for your patience!

RodrigoM · May 1, 2021, 2:13pm

Hi there,

Same problem here but this time with the latest versions of supervisor and balena CLI.

% balena push cc435b4.local --env ENVIRONMENT_SUFFIX=-dev --debug

[Debug]   Sending request to http://10.7.109.15:48484/v2/local/device-info
[Debug]   Found build tasks:
[Debug]       my-service: build [./services/my-service:]
[Debug]   Resolving services with [raspberrypi3-64|aarch64]
[Debug]   Found project types:
[Debug]       my-service:: Architecture-specific Dockerfile
[Debug]   Probing remote daemon for cache images
[Debug]   Using 79 on-device images for cache...
[Debug]   Starting builds...

Now Balena CLI hangs for about 3-4 minutes and only then it continues.

[Build]   [my-service:] Step 1/24 : FROM balenalib/aarch64-openjdk:11-jdk
[Build]   [my-service]  ---> 5b564b01a111
[Build]   [my-service] Step 2/24 : WORKDIR /home/my-service
[Build]   [my-service]  ---> Using cache
[Build]   [my-service]  ---> 6af095da957c

This is very annoying and slows down development considerably.
Any idea what is the cause? We have always had smooth builds until some days ago.

Current versions:

balena CLI (macOS): 12.44.12
balena Supervisor: 12.5.10
balena OS: 2.65.0+rev1 Raspberry Pi 3 (using 64bit OS)

Thanks

rahul-thakoor · May 7, 2021, 6:28am

Hey there,

Balena CLI hangs for about 3-4 minutes

Can you please clarify at what step the CLI hangs please?
Are you seeing the CLI hangs after each change?

Thanks

rahul-thakoor · May 7, 2021, 7:07am

I tried to reproduce this but couldn’t:

I’m using

Windows 10
balenaCLI (linux in wsl) : 12.44.12
balena Supervisor: 12.5.10

chrisys · May 14, 2021, 12:10pm

Hey there, just wondering if you got a chance to test this again and answer what my colleague asked above? There may be a delay during base image downloads which is to be expected but this shouldn’t happen for every code change.