Stuck on "Sending request to http://10.0.0.8:48484/v2/local/target-state" with balena push in local mode

What’s happening

I run balena push 10.0.0.8 and it works for the first time. My Express server runs and I’m able to see a “Hello world” message in 10.0.0.8:3000. If I immediately update the code to change the message that will be returned by Express, it works due to live push. But if I press Ctrl+C to detach from the logs, change the message returned by Express and try balena push 10.0.0.8 again, it hangs in Sending request to http://10.0.0.8:48484/v2/local/target-state and the changes are not applied so I keep seeing “Hello world” instead of the new message.

If I go to Balena Cloud, disable local mode, and then re-enable local mode, I’m able to push my code once again to the local device.

What I expected

To be able to run balena push 10.0.0.8 to update the code that’s running in my local device.

Versions

Host OS: balenaOS 2.53.12+rev1
Supervisor: 11.9.9
Device: Raspberry Pi 4 Model B
Balena CLI: 12.14.11

Logs:

[gustavo@sun balena]$ DEBUG=1 balena push 10.0.0.8
[debug] original argv0="node" argv=[/usr/local/bin/node,/usr/bin/balena,push,10.0.0.8] length=4
[debug] new argv=[/usr/local/bin/node,/usr/bin/balena,push,10.0.0.8] length=4
[debug] Using . as build source
[Debug]   Checking we can access device
[Debug]   Sending request to http://10.0.0.8:48484/ping
[Debug]   Checking device version: 11.9.9
[Info]    Starting build on device 10.0.0.8
[Debug]   Loading project...
[Debug]   Resolving project...
[Info]    No "docker-compose.yml" file found at "."
[Info]    Creating default composition with source: "."
[Debug]   Creating project...
[Debug]   Tarring all non-ignored files...
[Debug]   Sending request to http://10.0.0.8:48484/v2/local/device-info
[Debug]   Found build tasks:
[Debug]       main: build [.]
[Debug]   Resolving services with [raspberrypi4-64|aarch64]
[Debug]   Found project types:
[Debug]       main: Dockerfile.template
[Debug]   Probing remote daemon for cache images
[Debug]   Using 3 on-device images for cache...
[Debug]   Starting builds...
[Build]   [main] Step 1/9 : FROM balenalib/raspberrypi4-64-node:10
[Build]   [main]  ---> c99cec39d535
[Build]   [main] Step 2/9 : WORKDIR /usr/src/app
[Build]   [main]  ---> Running in 92a14961fcde
[Build]   [main] Removing intermediate container 92a14961fcde
[Build]   [main]  ---> 4bcd4a787594
[Build]   [main] Step 3/9 : COPY package.json package.json
[Build]   [main]  ---> 98e62b60a9c2
[Build]   [main] Step 4/9 : RUN JOBS=MAX npm install --production && npm cache verify && rm -rf /tmp/*
[Build]   [main]  ---> Running in fbec772e51af
[Build]   [main] Here are a few details about this Docker image (For more information please visit https://www.balena.io/docs/reference/base-images/base-images/): 
[Build]   Architecture: ARM v8 
[Build]   OS: Debian Buster 
[Build]   Variant: run variant 
[Build]   Default variable(s): UDEV=off 
[Build]   The following software stack is preinstalled: 
[Build]   Node.js v10.22.0, Yarn v1.22.4 
[Build]   Extra features: 
[Build]   - Easy way to install packages with `install_packages <package-name>` command 
[Build]   - Run anywhere with cross-build feature  (for ARM only) 
[Build]   - Keep the container idling with `balena-idle` command 
[Build]   - Show base image details with `balena-info` command
[Build]   [main] npm 
[Build]   [main] notice created a lockfile as package-lock.json. You should commit this file.
[Build]   
[Build]   [main] npm WARN
[Build]   [main]  app No description
[Build]   
[Build]   [main] npm WARN
[Build]   [main]  app No repository field.
[Build]   
[Build]   [main] npm WARN
[Build]   [main]  app No license field.
[Build]   
[Build]   [main] 
[Build]   
[Build]   [main] added 50 packages from 37 contributors and audited 50 packages in 4.979s
[Build]   [main] found 0 vulnerabilities
[Build]   [main] Cache verified and compressed (~/.npm/_cacache):
[Build]   [main] Content verified: 102 (1309809 bytes)
[Build]   [main] Index entries: 154
[Build]   [main] Finished in 0.905s
[Build]   [main] Removing intermediate container fbec772e51af
[Build]   [main]  ---> 38ea85428d36
[Build]   [main] Step 5/9 : COPY . .
[Build]   [main]  ---> ba3744a271f3
[Build]   [main] Step 6/9 : ENV UDEV=1
[Build]   [main]  ---> Running in 45935ccb8d81
[Build]   [main] Removing intermediate container 45935ccb8d81
[Build]   [main]  ---> 87d2c4b78b50
[Build]   [main] Step 7/9 : CMD ["npm", "start"]
[Build]   [main]  ---> Running in b2a8360c23c3
[Build]   [main] Removing intermediate container b2a8360c23c3
[Build]   [main]  ---> f481bf959e5c
[Build]   [main] Step 8/9 : LABEL io.resin.local.image=1
[Build]   [main]  ---> Running in aae55819f602
[Build]   [main] Removing intermediate container aae55819f602
[Build]   [main]  ---> 5e02d70ce2fc
[Build]   [main] Step 9/9 : LABEL io.resin.local.service=main
[Build]   [main]  ---> Running in 768ba2d98b5d
[Build]   [main] Removing intermediate container 768ba2d98b5d
[Build]   [main]  ---> d412ccbbfa19
[Build]   [main] Successfully built d412ccbbfa19
[Build]   [main] Successfully tagged local_image_main:latest
[Debug]   Setting device state...
[Debug]   Sending request to http://10.0.0.8:48484/v2/local/target-state
[Debug]   Sending target state: {"local":{"name":"empty-hill","config":{"SUPERVISOR_DELTA_VERSION":"3","HOST_CONFIG_arm_64bit":"1","HOST_CONFIG_disable_splash":"1","HOST_CONFIG_dtoverlay":"\"vc4-fkms-v3d\"","HOST_CONFIG_dtparam":"\"i2c_arm=on\",\"spi=on\",\"audio=on\"","HOST_CONFIG_enable_uart":"1","HOST_CONFIG_gpu_mem":"16","HOST_FIREWALL_MODE":"","SUPERVISOR_DELTA":"1","SUPERVISOR_LOCAL_MODE":"1","SUPERVISOR_POLL_INTERVAL":"900000","SUPERVISOR_VPN_CONTROL":"true","SUPERVISOR_INSTANT_UPDATE_TRIGGER":"true","SUPERVISOR_CONNECTIVITY_CHECK":"true","SUPERVISOR_LOG_CONTROL":"true","SUPERVISOR_DELTA_REQUEST_TIMEOUT":"30000","SUPERVISOR_DELTA_APPLY_TIMEOUT":"0","SUPERVISOR_DELTA_RETRY_COUNT":"30","SUPERVISOR_DELTA_RETRY_INTERVAL":"10000","SUPERVISOR_OVERRIDE_LOCK":"false","SUPERVISOR_PERSISTENT_LOGGING":"false","HOST_DISCOVERABILITY":"true"},"apps":{"1":{"name":"localapp","commit":"localrelease","releaseId":"1","services":{"1":{"environment":{},"labels":{"io.resin.features.kernel-modules":"1","io.resin.features.firmware":"1","io.resin.features.dbus":"1","io.resin.features.supervisor-api":"1","io.resin.features.resin-api":"1"},"privileged":true,"tty":true,"restart":"always","network_mode":"host","volumes":["resin-data:/data"],"imageId":1,"serviceName":"main","serviceId":1,"image":"local_image_main:latest","running":true}},"volumes":{"resin-data":{}},"networks":{}}}},"dependent":{"apps":[],"devices":[]}}
[Debug]   Sending request to http://10.0.0.8:48484/v2/local/target-state

[Debug]   Sending request to http://10.0.0.8:48484/v2/local/device-info
[Info]    Streaming device logs...
[Live]    Watching for file changes...
[Live]    Waiting for device state to settle...
[Logs]    [8/24/2020, 10:29:17 AM] Creating network 'default'
[Logs]    [8/24/2020, 10:29:17 AM] Creating volume 'resin-data'
[Debug]   Device state not settled, retrying in 1000ms
[Logs]    [8/24/2020, 10:29:17 AM] Installing service 'main sha256:d412ccbbfa19b4a4a2fae755edabe5c7edccc06e96efbcdc5552b68832f889d6'
[Logs]    [8/24/2020, 10:29:17 AM] Installed service 'main sha256:d412ccbbfa19b4a4a2fae755edabe5c7edccc06e96efbcdc5552b68832f889d6'
[Logs]    [8/24/2020, 10:29:17 AM] Starting service 'main sha256:d412ccbbfa19b4a4a2fae755edabe5c7edccc06e96efbcdc5552b68832f889d6'
[Logs]    [8/24/2020, 10:29:18 AM] Started service 'main sha256:d412ccbbfa19b4a4a2fae755edabe5c7edccc06e96efbcdc5552b68832f889d6'
[Debug]   Device state not settled, retrying in 1000ms
[Live]    Device state settled
[Logs]    [8/24/2020, 10:29:20 AM] [main] 
[Logs]    [8/24/2020, 10:29:20 AM] [main] > @ start /usr/src/app
[Logs]    [8/24/2020, 10:29:20 AM] [main] > node server.js
[Logs]    [8/24/2020, 10:29:20 AM] [main] 
[Logs]    [8/24/2020, 10:29:20 AM] [main] Listening on port 3000
^C[Live]    Cleaning up device...

[gustavo@sun balena]$ DEBUG=1 balena push 10.0.0.8
[debug] original argv0="node" argv=[/usr/local/bin/node,/usr/bin/balena,push,10.0.0.8] length=4
[debug] new argv=[/usr/local/bin/node,/usr/bin/balena,push,10.0.0.8] length=4
[debug] Using . as build source
[Debug]   Checking we can access device
[Debug]   Sending request to http://10.0.0.8:48484/ping
[Debug]   Checking device version: 11.9.9
[Info]    Starting build on device 10.0.0.8
[Debug]   Loading project...
[Debug]   Resolving project...
[Info]    No "docker-compose.yml" file found at "."
[Info]    Creating default composition with source: "."
[Debug]   Creating project...
[Debug]   Tarring all non-ignored files...
[Debug]   Sending request to http://10.0.0.8:48484/v2/local/device-info
[Debug]   Found build tasks:
[Debug]       main: build [.]
[Debug]   Resolving services with [raspberrypi4-64|aarch64]
[Debug]   Found project types:
[Debug]       main: Dockerfile.template
[Debug]   Probing remote daemon for cache images
[Debug]   Using 11 on-device images for cache...
[Debug]   Starting builds...
[Build]   [main] Step 1/9 : FROM balenalib/raspberrypi4-64-node:10
[Build]   [main]  ---> c99cec39d535
[Build]   [main] Step 2/9 : WORKDIR /usr/src/app
[Build]   [main]  ---> Using cache
[Build]   [main]  ---> 4bcd4a787594
[Build]   [main] Step 3/9 : COPY package.json package.json
[Build]   [main]  ---> Using cache
[Build]   [main]  ---> 98e62b60a9c2
[Build]   [main] Step 4/9 : RUN JOBS=MAX npm install --production && npm cache verify && rm -rf /tmp/*
[Build]   [main]  ---> Using cache
[Build]   [main]  ---> 38ea85428d36
[Build]   [main] Step 5/9 : COPY . .
[Build]   [main]  ---> 1c1e3dd3bf83
[Build]   [main] Step 6/9 : ENV UDEV=1
[Build]   [main]  ---> Running in 9b17ba10cfe9
[Build]   [main] Removing intermediate container 9b17ba10cfe9
[Build]   [main]  ---> 772bf71b7866
[Build]   [main] Step 7/9 : CMD ["npm", "start"]
[Build]   [main]  ---> Running in 7df41f05dca2
[Build]   [main] Removing intermediate container 7df41f05dca2
[Build]   [main]  ---> 12090331a679
[Build]   [main] Step 8/9 : LABEL io.resin.local.image=1
[Build]   [main]  ---> Running in cf94764f3f72
[Build]   [main] Removing intermediate container cf94764f3f72
[Build]   [main]  ---> 4b927f04239e
[Build]   [main] Step 9/9 : LABEL io.resin.local.service=main
[Build]   [main]  ---> Running in 408abf18266e
[Build]   [main] Removing intermediate container 408abf18266e
[Build]   [main]  ---> 28fed70f5bed
[Build]   [main] Successfully built 28fed70f5bed
[Build]   [main] Successfully tagged local_image_main:latest
[Debug]   Setting device state...
[Debug]   Sending request to http://10.0.0.8:48484/v2/local/target-state

Hey @gustavopch,

As you may have seen in the other thread, this is a known bug that was fixed in supervisor v11.11.4. We should be releasing an OS with a newer supervisor version soon, which will allow you to develop in localmode on the newest OS. Additionally, we are actively working on making supervisors easier to update without a full OS upgrade, so stay tuned for more information on that!

1 Like

Good to know! Just confirmed that pushing in local mode works as expected with balenaOS 2.51.1+rev1 and Supervisor 11.4.10.

Hi folks, I am on 12.2.11 and still getting stuck in “Probing remote daemon for cache images”. How does someone resolve this?

Hello,
Can you please tell us the context when you receive this error, and the full logs for us to gain relevant information to help you better regarding this. If it’s not relevant to the current thread, then I would recommend to open a new thread with your issue and with all the details. Our team would be happy to help you out. Thanks!

Thank you @vipulgupta2048; I’ve created a new topic here: Local mode push freezes after "Probing remote daemon for cache images"