Stuck on "Sending request to http://10.0.0.8:48484/v2/local/target-state" with balena push in local mode

What’s happening

I run balena push 10.0.0.8 and it works for the first time. My Express server runs and I’m able to see a “Hello world” message in 10.0.0.8:3000. If I immediately update the code to change the message that will be returned by Express, it works due to live push. But if I press Ctrl+C to detach from the logs, change the message returned by Express and try balena push 10.0.0.8 again, it hangs in Sending request to http://10.0.0.8:48484/v2/local/target-state and the changes are not applied so I keep seeing “Hello world” instead of the new message.

If I go to Balena Cloud, disable local mode, and then re-enable local mode, I’m able to push my code once again to the local device.

What I expected

To be able to run balena push 10.0.0.8 to update the code that’s running in my local device.

Versions

Host OS: balenaOS 2.53.12+rev1
Supervisor: 11.9.9
Device: Raspberry Pi 4 Model B
Balena CLI: 12.14.11

Logs:

[gustavo@sun balena]$ DEBUG=1 balena push 10.0.0.8
[debug] original argv0="node" argv=[/usr/local/bin/node,/usr/bin/balena,push,10.0.0.8] length=4
[debug] new argv=[/usr/local/bin/node,/usr/bin/balena,push,10.0.0.8] length=4
[debug] Using . as build source
[Debug]   Checking we can access device
[Debug]   Sending request to http://10.0.0.8:48484/ping
[Debug]   Checking device version: 11.9.9
[Info]    Starting build on device 10.0.0.8
[Debug]   Loading project...
[Debug]   Resolving project...
[Info]    No "docker-compose.yml" file found at "."
[Info]    Creating default composition with source: "."
[Debug]   Creating project...
[Debug]   Tarring all non-ignored files...
[Debug]   Sending request to http://10.0.0.8:48484/v2/local/device-info
[Debug]   Found build tasks:
[Debug]       main: build [.]
[Debug]   Resolving services with [raspberrypi4-64|aarch64]
[Debug]   Found project types:
[Debug]       main: Dockerfile.template
[Debug]   Probing remote daemon for cache images
[Debug]   Using 3 on-device images for cache...
[Debug]   Starting builds...
[Build]   [main] Step 1/9 : FROM balenalib/raspberrypi4-64-node:10
[Build]   [main]  ---> c99cec39d535
[Build]   [main] Step 2/9 : WORKDIR /usr/src/app
[Build]   [main]  ---> Running in 92a14961fcde
[Build]   [main] Removing intermediate container 92a14961fcde
[Build]   [main]  ---> 4bcd4a787594
[Build]   [main] Step 3/9 : COPY package.json package.json
[Build]   [main]  ---> 98e62b60a9c2
[Build]   [main] Step 4/9 : RUN JOBS=MAX npm install --production && npm cache verify && rm -rf /tmp/*
[Build]   [main]  ---> Running in fbec772e51af
[Build]   [main] Here are a few details about this Docker image (For more information please visit https://www.balena.io/docs/reference/base-images/base-images/): 
[Build]   Architecture: ARM v8 
[Build]   OS: Debian Buster 
[Build]   Variant: run variant 
[Build]   Default variable(s): UDEV=off 
[Build]   The following software stack is preinstalled: 
[Build]   Node.js v10.22.0, Yarn v1.22.4 
[Build]   Extra features: 
[Build]   - Easy way to install packages with `install_packages <package-name>` command 
[Build]   - Run anywhere with cross-build feature  (for ARM only) 
[Build]   - Keep the container idling with `balena-idle` command 
[Build]   - Show base image details with `balena-info` command
[Build]   [main] npm 
[Build]   [main] notice created a lockfile as package-lock.json. You should commit this file.
[Build]   
[Build]   [main] npm WARN
[Build]   [main]  app No description
[Build]   
[Build]   [main] npm WARN
[Build]   [main]  app No repository field.
[Build]   
[Build]   [main] npm WARN
[Build]   [main]  app No license field.
[Build]   
[Build]   [main] 
[Build]   
[Build]   [main] added 50 packages from 37 contributors and audited 50 packages in 4.979s
[Build]   [main] found 0 vulnerabilities
[Build]   [main] Cache verified and compressed (~/.npm/_cacache):
[Build]   [main] Content verified: 102 (1309809 bytes)
[Build]   [main] Index entries: 154
[Build]   [main] Finished in 0.905s
[Build]   [main] Removing intermediate container fbec772e51af
[Build]   [main]  ---> 38ea85428d36
[Build]   [main] Step 5/9 : COPY . .
[Build]   [main]  ---> ba3744a271f3
[Build]   [main] Step 6/9 : ENV UDEV=1
[Build]   [main]  ---> Running in 45935ccb8d81
[Build]   [main] Removing intermediate container 45935ccb8d81
[Build]   [main]  ---> 87d2c4b78b50
[Build]   [main] Step 7/9 : CMD ["npm", "start"]
[Build]   [main]  ---> Running in b2a8360c23c3
[Build]   [main] Removing intermediate container b2a8360c23c3
[Build]   [main]  ---> f481bf959e5c
[Build]   [main] Step 8/9 : LABEL io.resin.local.image=1
[Build]   [main]  ---> Running in aae55819f602
[Build]   [main] Removing intermediate container aae55819f602
[Build]   [main]  ---> 5e02d70ce2fc
[Build]   [main] Step 9/9 : LABEL io.resin.local.service=main
[Build]   [main]  ---> Running in 768ba2d98b5d
[Build]   [main] Removing intermediate container 768ba2d98b5d
[Build]   [main]  ---> d412ccbbfa19
[Build]   [main] Successfully built d412ccbbfa19
[Build]   [main] Successfully tagged local_image_main:latest
[Debug]   Setting device state...
[Debug]   Sending request to http://10.0.0.8:48484/v2/local/target-state
[Debug]   Sending target state: {"local":{"name":"empty-hill","config":{"SUPERVISOR_DELTA_VERSION":"3","HOST_CONFIG_arm_64bit":"1","HOST_CONFIG_disable_splash":"1","HOST_CONFIG_dtoverlay":"\"vc4-fkms-v3d\"","HOST_CONFIG_dtparam":"\"i2c_arm=on\",\"spi=on\",\"audio=on\"","HOST_CONFIG_enable_uart":"1","HOST_CONFIG_gpu_mem":"16","HOST_FIREWALL_MODE":"","SUPERVISOR_DELTA":"1","SUPERVISOR_LOCAL_MODE":"1","SUPERVISOR_POLL_INTERVAL":"900000","SUPERVISOR_VPN_CONTROL":"true","SUPERVISOR_INSTANT_UPDATE_TRIGGER":"true","SUPERVISOR_CONNECTIVITY_CHECK":"true","SUPERVISOR_LOG_CONTROL":"true","SUPERVISOR_DELTA_REQUEST_TIMEOUT":"30000","SUPERVISOR_DELTA_APPLY_TIMEOUT":"0","SUPERVISOR_DELTA_RETRY_COUNT":"30","SUPERVISOR_DELTA_RETRY_INTERVAL":"10000","SUPERVISOR_OVERRIDE_LOCK":"false","SUPERVISOR_PERSISTENT_LOGGING":"false","HOST_DISCOVERABILITY":"true"},"apps":{"1":{"name":"localapp","commit":"localrelease","releaseId":"1","services":{"1":{"environment":{},"labels":{"io.resin.features.kernel-modules":"1","io.resin.features.firmware":"1","io.resin.features.dbus":"1","io.resin.features.supervisor-api":"1","io.resin.features.resin-api":"1"},"privileged":true,"tty":true,"restart":"always","network_mode":"host","volumes":["resin-data:/data"],"imageId":1,"serviceName":"main","serviceId":1,"image":"local_image_main:latest","running":true}},"volumes":{"resin-data":{}},"networks":{}}}},"dependent":{"apps":[],"devices":[]}}
[Debug]   Sending request to http://10.0.0.8:48484/v2/local/target-state

[Debug]   Sending request to http://10.0.0.8:48484/v2/local/device-info
[Info]    Streaming device logs...
[Live]    Watching for file changes...
[Live]    Waiting for device state to settle...
[Logs]    [8/24/2020, 10:29:17 AM] Creating network 'default'
[Logs]    [8/24/2020, 10:29:17 AM] Creating volume 'resin-data'
[Debug]   Device state not settled, retrying in 1000ms
[Logs]    [8/24/2020, 10:29:17 AM] Installing service 'main sha256:d412ccbbfa19b4a4a2fae755edabe5c7edccc06e96efbcdc5552b68832f889d6'
[Logs]    [8/24/2020, 10:29:17 AM] Installed service 'main sha256:d412ccbbfa19b4a4a2fae755edabe5c7edccc06e96efbcdc5552b68832f889d6'
[Logs]    [8/24/2020, 10:29:17 AM] Starting service 'main sha256:d412ccbbfa19b4a4a2fae755edabe5c7edccc06e96efbcdc5552b68832f889d6'
[Logs]    [8/24/2020, 10:29:18 AM] Started service 'main sha256:d412ccbbfa19b4a4a2fae755edabe5c7edccc06e96efbcdc5552b68832f889d6'
[Debug]   Device state not settled, retrying in 1000ms
[Live]    Device state settled
[Logs]    [8/24/2020, 10:29:20 AM] [main] 
[Logs]    [8/24/2020, 10:29:20 AM] [main] > @ start /usr/src/app
[Logs]    [8/24/2020, 10:29:20 AM] [main] > node server.js
[Logs]    [8/24/2020, 10:29:20 AM] [main] 
[Logs]    [8/24/2020, 10:29:20 AM] [main] Listening on port 3000
^C[Live]    Cleaning up device...

[gustavo@sun balena]$ DEBUG=1 balena push 10.0.0.8
[debug] original argv0="node" argv=[/usr/local/bin/node,/usr/bin/balena,push,10.0.0.8] length=4
[debug] new argv=[/usr/local/bin/node,/usr/bin/balena,push,10.0.0.8] length=4
[debug] Using . as build source
[Debug]   Checking we can access device
[Debug]   Sending request to http://10.0.0.8:48484/ping
[Debug]   Checking device version: 11.9.9
[Info]    Starting build on device 10.0.0.8
[Debug]   Loading project...
[Debug]   Resolving project...
[Info]    No "docker-compose.yml" file found at "."
[Info]    Creating default composition with source: "."
[Debug]   Creating project...
[Debug]   Tarring all non-ignored files...
[Debug]   Sending request to http://10.0.0.8:48484/v2/local/device-info
[Debug]   Found build tasks:
[Debug]       main: build [.]
[Debug]   Resolving services with [raspberrypi4-64|aarch64]
[Debug]   Found project types:
[Debug]       main: Dockerfile.template
[Debug]   Probing remote daemon for cache images
[Debug]   Using 11 on-device images for cache...
[Debug]   Starting builds...
[Build]   [main] Step 1/9 : FROM balenalib/raspberrypi4-64-node:10
[Build]   [main]  ---> c99cec39d535
[Build]   [main] Step 2/9 : WORKDIR /usr/src/app
[Build]   [main]  ---> Using cache
[Build]   [main]  ---> 4bcd4a787594
[Build]   [main] Step 3/9 : COPY package.json package.json
[Build]   [main]  ---> Using cache
[Build]   [main]  ---> 98e62b60a9c2
[Build]   [main] Step 4/9 : RUN JOBS=MAX npm install --production && npm cache verify && rm -rf /tmp/*
[Build]   [main]  ---> Using cache
[Build]   [main]  ---> 38ea85428d36
[Build]   [main] Step 5/9 : COPY . .
[Build]   [main]  ---> 1c1e3dd3bf83
[Build]   [main] Step 6/9 : ENV UDEV=1
[Build]   [main]  ---> Running in 9b17ba10cfe9
[Build]   [main] Removing intermediate container 9b17ba10cfe9
[Build]   [main]  ---> 772bf71b7866
[Build]   [main] Step 7/9 : CMD ["npm", "start"]
[Build]   [main]  ---> Running in 7df41f05dca2
[Build]   [main] Removing intermediate container 7df41f05dca2
[Build]   [main]  ---> 12090331a679
[Build]   [main] Step 8/9 : LABEL io.resin.local.image=1
[Build]   [main]  ---> Running in cf94764f3f72
[Build]   [main] Removing intermediate container cf94764f3f72
[Build]   [main]  ---> 4b927f04239e
[Build]   [main] Step 9/9 : LABEL io.resin.local.service=main
[Build]   [main]  ---> Running in 408abf18266e
[Build]   [main] Removing intermediate container 408abf18266e
[Build]   [main]  ---> 28fed70f5bed
[Build]   [main] Successfully built 28fed70f5bed
[Build]   [main] Successfully tagged local_image_main:latest
[Debug]   Setting device state...
[Debug]   Sending request to http://10.0.0.8:48484/v2/local/target-state

Hey @gustavopch,

As you may have seen in the other thread, this is a known bug that was fixed in supervisor v11.11.4. We should be releasing an OS with a newer supervisor version soon, which will allow you to develop in localmode on the newest OS. Additionally, we are actively working on making supervisors easier to update without a full OS upgrade, so stay tuned for more information on that!

1 Like

Good to know! Just confirmed that pushing in local mode works as expected with balenaOS 2.51.1+rev1 and Supervisor 11.4.10.