Can't deploy new code

I have a docker-compose.yml file that contains multiple services.

If I execute balena deploy <APP_NAME> --logs --source . -b, I can see that the project was built successfully.
But if I check the device status, its commit value doesn’t change. So seems like it is not updated at all.

root@plextreme-UP-WHL01:~/BalenaPOC# balena apps
[debug] original argv0="node" argv=[/usr/bin/node,/usr/bin/balena,apps] length=3
ID APP NAME DEVICE TYPE ONLINE DEVICES DEVICE COUNT
2  testPOC  intel-nuc   1              1
root@plextreme-UP-WHL01:~/BalenaPOC# balena devices
[debug] original argv0="node" argv=[/usr/bin/node,/usr/bin/balena,devices] length=3
ID UUID    DEVICE NAME     DEVICE TYPE APPLICATION NAME STATUS IS ONLINE SUPERVISOR VERSION OS VERSION           DASHBOARD URL
1  7c21238 fragrant-meadow intel-nuc   testPOC          Idle   true      10.3.7             balenaOS 2.44.0+rev1 https://dashboard.viso.dev/devices/7c21238b2db06050822b91aa5e97327b/summary
root@plextreme-UP-WHL01:~/BalenaPOC# balena device 7c21238
[debug] original argv0="node" argv=[/usr/bin/node,/usr/bin/balena,device,7c21238] length=4
== FRAGRANT MEADOW
ID:                 1
DEVICE TYPE:        intel-nuc
STATUS:             inactive
IS ONLINE:          true
IP ADDRESS:         192.168.178.53
APPLICATION NAME:   testPOC
UUID:               7c21238b2db06050822b91aa5e97327b
COMMIT:             1a3ef4812b54c44960e148af5cec4070
SUPERVISOR VERSION: 10.3.7
OS VERSION:         balenaOS 2.44.0+rev1
DASHBOARD URL:      https://dashboard.viso.dev/devices/7c21238b2db06050822b91aa5e97327b/summary

Any idea?

The COMMIT value(1a3ef4812b54c44960e148af5cec4070) is not updated though I build the project for several times… :confused:

Hi, Could you have a look at the supervisor logs of that device. If you have access to the hostOS of the device you can check these logs by doing journalctl -u resin-supervisor --no-pager . It should be clear from there is the supervisor is having some kind of issue.

1 Like

@shaunmulligan
I just noticed that the balena deploy changes the releaseHash of my app.
I’ve found this tool - https://github.com/balena-io-projects/staged-releases/blob/master/pin-devices-running-release.sh

But how can I get the environment variables in balena.env?

export BASE_URL=viso.dev
export APP_ID=testPOC     <<<<< App Name? or App Id?
export authToken=<PUT YOUR AUTH TOKEN HERE>

Especially authToken?

Cheers

If deploy is changing the releaseHash of the App, that means the deploy part is working properly but the device is not pulling the image, so pulling the supervisor logs will help here. The staged-release project won’t help with this problem as it sounds like your device is not pulling releases at all, so pinning it to specific releases won’t help.

In any case, if you would like to use the staged release project once you have fixed your device, you can find the App ID by running balena apps and you can get the Auth token by running 'balena api-key generate “My staged release key”

1 Like

Okay, thanks.
Here is the output on the hostOS:

-- Logs begin at Wed 2019-11-27 06:30:30 UTC, end at Wed 2019-11-27 08:56:02 UTC. --
Nov 27 06:30:58 7c21238 resin-supervisor[1452]: resin_supervisor
Nov 27 06:30:58 7c21238 resin-supervisor[1496]: active
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: Container config has not changed
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: Starting system message bus: dbus.
Nov 27 06:30:59 7c21238 resin-supervisor[1497]:  * Starting Avahi mDNS/DNS-SD Daemon: avahi-daemon
Nov 27 06:30:59 7c21238 resin-supervisor[1497]:    ...done.
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [info]    Supervisor v10.3.7 starting up...
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: (node:1) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [debug]   Starting event tracker
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [debug]   Starting api binder
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [debug]   Starting logging infrastructure
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [event]   Event: Supervisor start {}
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [debug]   Performing database cleanup for container log timestamps
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [info]    Previous engine snapshot was not stored. Skipping cleanup.
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [debug]   Handling of local mode switch is completed
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [debug]   Connectivity check enabled: true
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [debug]   Starting periodic check for IP addresses
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [info]    Reporting initial state, supervisor version and API info
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [debug]   VPN status path exists.
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [info]    Waiting for connectivity...
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [debug]   Skipping preloading
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [info]    Starting API server
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [info]    Applying target state
Nov 27 06:30:59 7c21238 resin-supervisor[1497]: [debug]   Ensuring device is provisioned
Nov 27 06:31:00 7c21238 resin-supervisor[1497]: [debug]   Starting current state report
Nov 27 06:31:00 7c21238 resin-supervisor[1497]: [debug]   Starting target state poll
Nov 27 06:31:00 7c21238 resin-supervisor[1497]: [debug]   Supervisor API listening on allowed interfaces only
Nov 27 06:31:00 7c21238 resin-supervisor[1497]: [debug]   Finished applying target state
Nov 27 06:31:00 7c21238 resin-supervisor[1497]: [success] Device state apply success
Nov 27 06:31:00 7c21238 resin-supervisor[1497]: [info]    Applying target state
Nov 27 06:31:00 7c21238 resin-supervisor[1497]: [debug]   Finished applying target state
Nov 27 06:31:00 7c21238 resin-supervisor[1497]: [success] Device state apply success
Nov 27 06:31:09 7c21238 resin-supervisor[1497]: [info]    Internet Connectivity: OK
Nov 27 06:31:50 7c21238 resin-supervisor[1497]: [error]   LogBackend: server responded with status code: 504
Nov 27 06:35:59 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 3.511 ms
Nov 27 06:36:47 7c21238 resin-supervisor[1497]: [event]   Event: Update notification {}
Nov 27 06:36:47 7c21238 resin-supervisor[1497]: [api]     POST /v1/update 204 - 5.163 ms
Nov 27 06:36:48 7c21238 resin-supervisor[1497]: [info]    Applying target state
Nov 27 06:36:48 7c21238 resin-supervisor[1497]: [debug]   Finished applying target state
Nov 27 06:36:48 7c21238 resin-supervisor[1497]: [success] Device state apply success
Nov 27 06:40:59 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 1.682 ms
Nov 27 06:40:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 06:40:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 06:45:59 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.875 ms
Nov 27 06:50:59 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 1.832 ms
Nov 27 06:50:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 06:50:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 06:55:59 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.831 ms
Nov 27 07:00:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 07:00:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 07:00:59 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.873 ms
Nov 27 07:06:00 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 1.159 ms
Nov 27 07:10:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 07:10:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 07:11:00 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.860 ms
Nov 27 07:16:00 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 1.030 ms
Nov 27 07:20:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 07:20:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 07:21:00 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 1.116 ms
Nov 27 07:22:08 7c21238 resin-supervisor[1497]: [event]   Event: Update notification {}
Nov 27 07:22:08 7c21238 resin-supervisor[1497]: [api]     POST /v1/update 204 - 6.214 ms
Nov 27 07:22:08 7c21238 resin-supervisor[1497]: [info]    Applying target state
Nov 27 07:22:08 7c21238 resin-supervisor[1497]: [debug]   Finished applying target state
Nov 27 07:22:08 7c21238 resin-supervisor[1497]: [success] Device state apply success
Nov 27 07:22:38 7c21238 resin-supervisor[1497]: [event]   Event: Update notification {}
Nov 27 07:22:38 7c21238 resin-supervisor[1497]: [api]     POST /v1/update 204 - 5.381 ms
Nov 27 07:22:38 7c21238 resin-supervisor[1497]: [info]    Applying target state
Nov 27 07:22:38 7c21238 resin-supervisor[1497]: [debug]   Finished applying target state
Nov 27 07:22:38 7c21238 resin-supervisor[1497]: [success] Device state apply success
Nov 27 07:26:00 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.796 ms
Nov 27 07:30:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 07:30:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 07:31:00 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.755 ms
Nov 27 07:36:00 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.781 ms
Nov 27 07:40:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 07:40:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 07:41:00 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.856 ms
Nov 27 07:46:00 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.764 ms
Nov 27 07:50:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 07:50:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 07:51:01 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.769 ms
Nov 27 07:56:01 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 1.397 ms
Nov 27 08:00:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 08:00:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 08:01:01 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 1.067 ms
Nov 27 08:06:01 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.783 ms
Nov 27 08:10:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 08:10:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 08:11:01 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 1.417 ms
Nov 27 08:16:01 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 1.180 ms
Nov 27 08:20:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 08:20:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 08:21:01 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.722 ms
Nov 27 08:26:01 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.740 ms
Nov 27 08:30:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 08:30:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 08:31:01 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.739 ms
Nov 27 08:36:02 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.787 ms
Nov 27 08:40:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 08:40:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 08:41:02 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.934 ms
Nov 27 08:44:17 7c21238 resin-supervisor[1497]: [event]   Event: Update notification {}
Nov 27 08:44:17 7c21238 resin-supervisor[1497]: [api]     POST /v1/update 204 - 5.746 ms
Nov 27 08:44:18 7c21238 resin-supervisor[1497]: [info]    Applying target state
Nov 27 08:44:18 7c21238 resin-supervisor[1497]: [debug]   Finished applying target state
Nov 27 08:44:18 7c21238 resin-supervisor[1497]: [success] Device state apply success
Nov 27 08:46:02 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.743 ms
Nov 27 08:50:59 7c21238 resin-supervisor[1497]: [debug]   Attempting container log timestamp flush...
Nov 27 08:50:59 7c21238 resin-supervisor[1497]: [debug]   Container log timestamp flush complete
Nov 27 08:51:02 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 1.119 ms
Nov 27 08:56:02 7c21238 resin-supervisor[1497]: [api]     GET /v1/healthy 200 - 0.728 ms
root@7c21238:~# 

Cheers.

Okay, that is interesting. It seems the device was never notified of the change in releaseHash. When you did the balena deploy did you change any code? also did you add the --build flag to the command to force a rebuild. It might be that since nothing has changed (i.e. the same container images were uploaded each time) it didn’t actually need to update anything, so not update occured

Of course, I had changed my code… lol
This is my command:

balena deploy testPOC --nocache -b -s ./

Anything wrong? :slight_smile:
And I’ve just found another issue… :confused:
If we execute balena deploy, this will be built on the machine that I’ve executed.

The target device has GPU board, so I have to install nvidia driver.
But the local PC where the balena deploy command is executed does not have the gpu board…
So I am getting this error when building:

root@plextreme-UP-WHL01:~/BalenaPOC# balena deploy testPOC --nocache -b -s ./
[Info]    Compose file detected
[Info]    Building for amd64/intel-nuc
[Build]   Built 5 services in 2:47
[Build]   redis           Image is up to date for redis:latest
[Build]   mqtt            Image is up to date for eclipse-mosquitto:latest
[Build]   node-red        Successfully tagged balenapoc_node-red:latest
[Build]   object-detector [==========>          ]  50% Step 11/22: 2 upgraded, 61 newly installed, 0 to remove and 45 not upgraded.
[Build]   camera-feeder   Successfully tagged balenapoc_camera-feeder:latest
[Error]   Deploy failed
The command '/bin/bash -c apt-get install -y nvidia-driver-435' returned a non-zero code: 100

Additional information may be available by setting a DEBUG=1 environment
variable: "set DEBUG=1" on a Windows command prompt, or "export DEBUG=1"
on Linux or macOS.

If you need help, don't hesitate in contacting our support forums at
https://forums.balena.io

For bug reports or feature requests, have a look at the GitHub issues or
create a new one at: https://github.com/balena-io/balena-cli/issues/

Do I have to execute balena deploy command on the one with GPU?

Cheers.

haha, I have to ask these types of questions sometimes just to make sure I know all the context :slight_smile:

The command looks correct, i can’t see why it wouldn’t be working for you. I will ping some of the openBalena devs to come and see if they can see soemthing I am missing.

With GPU issue, i think its more likely that your container is trying to build for the wrong arch (x86) so i think what you would want to do is pass the --emulated flag to the deploy command, so that it knows to build for armv8 rather than x86.

Hmm, both have Intel CPUs, not Arm? lol

Oh, sorry I didn’t realise that. Hmm thats very odd then, because during buildtime you can’t have access to gpu devices, docker build containers are completely isolated so not sure how one would get around that…

Doing a bit of a google, it seems that the 100 error from apt-get is related to not having apt-transport-https installed in the image, could you run:

apt-get install -y apt-transport-https

before the install of that package. Found this in: https://stackoverflow.com/questions/38002543/apt-get-update-returned-a-non-zero-code-100

1 Like

One other thought I had was to test that your update notifications are correctly going to the device, could you try setting an env var or issuing the device identify <uuid> command and tailing the supervisor logs ( journalctl -u resin-supervisor -f). If the notification service is working correctly, you should see logs written out stating that a blink or configuration change was triggered

Also it would help to do the following:

curl -Ss https://api.<openbalena domain>/v5/device -H "Authorization: Bearer <api key>" | jq .

and

curl -Ss https://api.<openbalena domain>/v5/application -H "Authorization: Bearer <api key>" | jq .

replacing your domain and api key, in the command. We can then see if the device is perhaps somehow pinned.

Thanks for your patience, @shaunmulligan
It’s midnight here, so I will give it a try tomorrow morning early(maybe after 8 hours)!

Thanks a lot, mate! :+1:

No problem, we will be here once you have time to try some of these.

@shaunmulligan

Mate, how have you been?

Now I am getting another strange error:

image

It got stuck on camera-feeder container.
But this is the content of camera-feeder/Dockerfile.template file:

image

Pretty simple, just a normal python app…
Any idea?

Cheers,
Shane.

Cleaned local docker files from /var/lib/docker/overlay2, and it work now… :slight_smile:

@shaunmulligan
The result image files are pretty large…
Seems this prevents the automatic updating?

image

Did you simply delete everything inside /var/lib/docker/overlay2 folder? If so, did that have any effects on the running services?