Latest commit not deploying

patrick · January 26, 2019, 4:43pm

Should the balena app commit # be the same as the deploy release?

My latest commit for a multi-container project is not getting pushed to devices.
Preloading the project works (but then gets overwritten with what open balena believes is latest.)

$ balena deploy catch-connect-rpi3 --logs --source . --emulated --build --force
[Info]    Compose file detected
[Info]    Building for armv7hf/raspberrypi3
[Build]   Building services...
[Build]   catch-connect Preparing...
[Build]   display       Preparing...
[Build]   display       Step 1/6 : FROM petrosagg/resin-wpe:raspberrypi3-30c7465
[Build]   display        ---> 4b90e1c6a06a
[Build]   display       Step 2/6 : COPY udev-rules/ /etc/udev/rules.d/
[Build]   display        ---> Using cache
[Build]   display        ---> c0e6e3480b41
[Build]   display       Step 3/6 : COPY wpe-init /wpe-init
[Build]   display        ---> Using cache
[Build]   display        ---> d56a9a4392e4
[Build]   display       Step 4/6 : COPY welcome /var/lib/welcome
[Build]   display        ---> Using cache
[Build]   display        ---> 5459b667ccdd
[Build]   display       Step 5/6 : ENV WPE_URL="file:///var/lib/welcome/index.html"
[Build]   display        ---> Using cache
[Build]   display        ---> ab259a370b1a
[Build]   display       Step 6/6 : CMD [ "/wpe-init" ]
[Build]   display        ---> Using cache
[Build]   display        ---> c07ec8a5dfe4
[Build]   display       Successfully built c07ec8a5dfe4
[Build]   display       Successfully tagged catch-connect-rpi3_display:latest
[Info]    Creating release...
[Info]    Pushing images to registry...
[Info]    Saving release...
[Success] Deploy succeeded!
[Success] Release: a53f0185112dfce6ea02ef5d01bc5ca0

Then checking it with:
$ balena app 1
== CATCH CONNECT RPI3
ID: 1
DEVICE TYPE: raspberrypi3
COMMIT: 3792751532d6788ddb6505f7f0b91d09

Balena OS Raspberry Pi 2.29.2-rev1
Open Balena 0.1.4
Balena cli 9.12.0

patrick · January 27, 2019, 2:22pm

Just for fun I deploy the exact same project 24 hours later. This time the catch-connect container showed all of the build steps instead of stopping after Preparing...

Why would the deploy behave differently?

This release still does not get pushed to devices.
Does it have something to do with the display container not being tagged as latest?

$ balena deploy catch-connect-rpi3 --logs --source . --emulated --build

[Info] Compose file detected

[Info] Building for armv7hf/raspberrypi3

[Build] Building services...

[Build] **catch-connect** Preparing...

[Build] **display** Preparing...

[Build] **display** Step 1/6 : FROM petrosagg/resin-wpe:raspberrypi3-30c7465

[Build] **display** ---&gt; 4b90e1c6a06a

[Build] **display** Step 2/6 : COPY udev-rules/ /etc/udev/rules.d/

[Build] **display** ---&gt; Using cache

[Build] **display** ---&gt; c0e6e3480b41

[Build] **catch-connect** Step 1/3 : FROM resin/raspberrypi3-node:8

[Build] **catch-connect** ---&gt; 7b2a40165988

[Build] **catch-connect** Step 2/3 : COPY catch-connect-client /catch-connect-client

[Build] **display** Step 3/6 : COPY wpe-init /wpe-init

[Build] **display** ---&gt; Using cache

[Build] **display** ---&gt; d56a9a4392e4

[Build] **display** Step 4/6 : COPY welcome /var/lib/welcome

[Build] **display** ---&gt; Using cache

[Build] **display** ---&gt; 5459b667ccdd

[Build] **display** Step 5/6 : ENV WPE_URL="file:///var/lib/welcome/index.html"

[Build] **catch-connect** ---&gt; Using cache

[Build] **catch-connect** ---&gt; 0d607bd410b5

[Build] **catch-connect** Step 3/3 : CMD ["/catch-connect-client"]

[Build] **catch-connect** ---&gt; Using cache

[Build] **catch-connect** ---&gt; bb731f2ebdf6

[Build] **catch-connect** Successfully built bb731f2ebdf6

[Build] **catch-connect** Successfully tagged catch-connect-rpi3_catch-connect:latest

[Info] Creating release...

[Info] Pushing images to registry...

[Info] Saving release...

[Success] Deploy succeeded!

[Success] Release: 9f9126f44eab0a0757e6b642ecf9df05

richbayliss · January 29, 2019, 1:45pm

Hey @patrick

That indeed sounds a bit weird. Could you provide your compose file for starters, just so there is some context to work with.

Cheers.

patrick · January 29, 2019, 2:20pm

Thanks for having a look @richbayliss

Here’s my docker-compose.yml (the indenting is screwy on the forum):

version: '2.1'
volumes: 
  catch-connect-client-data:
services:
  catch-connect:
    build: ./catch-connect
    privileged: true
    restart: always
    volumes:
      - 'catch-connect-client-data:/catch-connect-data'
    depends_on:
      - display
    expose:
      - "443"
  display:
    build: ./wpe
    devices:
      - "/dev/fb0:/dev/fb0"
      - "/dev/vchiq:/dev/vchiq" 
    restart: always

richbayliss · January 30, 2019, 1:32pm

@patrick thanks for this.

So a few things to make clear and keep in mind;

the build process is happening on your workstation, not in openbalena, so a failure here could be related to your setup.
the device will only be “pushed” an update if the VPN is connected, or once it connects back to the openbalena server to checkin (approx. every 10mins)
the openbalena stack is still not considered production-ready and we cannot recommend it for a production deployment.

Could you confirm the following:

when your openbalena instance was last updated, so we can get an idea of the service versions you’re running
that you have deployed any application successfully with this openbalena instance, or is this your first attempt
the output of the following command:
- balena devices

Sorry if this takes a while to resolve, there are a lot of moving parts.

Thanks.

patrick · January 30, 2019, 1:44pm

@richbayliss Thanks for responding…

Blockquote * when your openbalena instance was last updated, so we can get an idea of the service versions you’re running*

updated last week
Balena OS Raspberry Pi 2.29.2-rev1
Open Balena 0.1.4
Balena cli 9.12.0

Blockquote * that you have deployed any application successfully with this openbalena instance, or is this your first attempt*

We’ve used the instance before and it worked as expected.

Blockquote * the output of the following command:
balena devices*

$ balena devices ID UUID DEVICE NAME DEVICE TYPE APPLICATION NAME STATUS IS ONLINE SUPERVISOR VERSION OS VERSION DASHBOARD URL 4 8a0d86c aged-butterfly raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.0+rev1 https://dashboard.balena.catchtechnologies.com/devices/8a0d86c495ead1468d3f18502265017a/summary 5 69ff140 black-meadow raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.0+rev1 https://dashboard.balena.catchtechnologies.com/devices/69ff140438e4db92e7d1c5f5e60a2e4f/summary 25 b9177d4 falling-morning raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.2+rev1 https://dashboard.balena.catchtechnologies.com/devices/b9177d4652169750aee68e5fc2390811/summary 8 fb63101 muddy-cloud raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.0+rev1 https://dashboard.balena.catchtechnologies.com/devices/fb6310168747fd29f235fb2aaa5716f1/summary 31 b947ff1 old-fog raspberrypi3 catch-connect-rpi3 Idle true 9.0.1 balenaOS 2.29.2+rev1 https://dashboard.balena.catchtechnologies.com/devices/b947ff12b345db61661dc554280cbb7a/summary 22 b372134 polished-shape raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.2+rev1 https://dashboard.balena.catchtechnologies.com/devices/b372134dadf41601e2a23a2e49857c37/summary 30 73b6938 quiet-silence raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.2+rev1 https://dashboard.balena.catchtechnologies.com/devices/73b6938b61036873cf217ed6fa591535/summary 26 b136752 rough-lake raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.2+rev1 https://dashboard.balena.catchtechnologies.com/devices/b13675296cb9040ee6bdf572d7f52447/summary 9 30bc85f shy-dew raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.0+rev1 https://dashboard.balena.catchtechnologies.com/devices/30bc85f2b62a3fa019baa316f8831a7c/summary 17 72a9a2b small-snow raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.2+rev1 https://dashboard.balena.catchtechnologies.com/devices/72a9a2b618bfaf9ccc15193b746634fc/summary 27 fe56b62 solitary-field raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.2+rev1 https://dashboard.balena.catchtechnologies.com/devices/fe56b6252bfc180abd05c271ef08da37/summary 20 f2b090d summer-morning raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.2+rev1 https://dashboard.balena.catchtechnologies.com/devices/f2b090d90729b7fed5135226b5167ba6/summary 24 09cf86a winter-lake raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.2+rev1 https://dashboard.balena.catchtechnologies.com/devices/09cf86a0040a3358a57e78443e1b2315/summary 23 18e6c51 wispy-breeze raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.2+rev1 https://dashboard.balena.catchtechnologies.com/devices/18e6c51ea54db1df45649348bc3e09f6/summary 21 4e7da97 wispy-sky raspberrypi3 catch-connect-rpi3 Idle false 9.0.1 balenaOS 2.29.2+rev1 https://dashboard.balena.catchtechnologies.com/devices/4e7da97c30d7a9c582c79cff14c6f5d6/summary

richbayliss · January 30, 2019, 3:22pm

@patrick

I have done some testing here;

setup a totally fresh instance of openbalena
added a new Pi 3B+ device
confirmed it came online
created a test app “test-app”
deployed a simple multicontainer application to “test-app”
watched the supervisor logs to see it get triggered straight away
confirmed it is operational

Looking at the device list you sent over (thanks, btw) it showed only a single device as online, so I will assume you’re testing with that one. I think the next step will be to get some ideas from the logs of that device, so if you SSH into that Pi and run journalctl -u resin-supervisor -fn100 you should see the last 100 or so lines of supervisor logs. If you leave it running then it will stream in any new log entries.

If you do this, then do another deploy then see what happens in the supervisor logs after it completes. I saw it instantly start to pull the services and start them.

See if this shows anything useful and post back here if you spot anything that looks weird.

Thanks.

richbayliss · January 30, 2019, 4:53pm

@patrick also just to answer your earlier question about release/commit IDs; when you do a balena deploy operation you should be presented with a release value. This is held on the device record as commit and accessible via the Supervisor API where it is also commit. See the screenshot attached for this, highlighted in red.

FYI: the web page shown is my test app, running on the device, and using the Supervisor API to acquire the values.

patrick · January 30, 2019, 5:07pm

@richbayliss Thanks for all the effort.

SSH is only for dev images, right?

Here is the cli log for that device:


30.01.19 14:50:39 (+0100) Service is already running 'catch-connect sha256:4e44e680cd136cbf8abd2ef62583fe6b9101243dbb114bc0454ca648ca634748'

30.01.19 14:50:39 (+0100) Service is already running 'display sha256:c07ec8a5dfe4314a83a9e246643f21726dfc41c9b6fe0c9cd2dff0cb5a20ac08'

30.01.19 14:55:53 (+0100) Killing service 'catch-connect sha256:4e44e680cd136cbf8abd2ef62583fe6b9101243dbb114bc0454ca648ca634748'

30.01.19 14:55:53 (+0100) Killing service 'display sha256:c07ec8a5dfe4314a83a9e246643f21726dfc41c9b6fe0c9cd2dff0cb5a20ac08'

30.01.19 14:55:58 (+0100) Service exited 'catch-connect sha256:4e44e680cd136cbf8abd2ef62583fe6b9101243dbb114bc0454ca648ca634748'

30.01.19 14:56:02 (+0100) Killed service 'catch-connect sha256:4e44e680cd136cbf8abd2ef62583fe6b9101243dbb114bc0454ca648ca634748'

30.01.19 14:56:03 (+0100) Deleting image 'registry.balena.catchtechnologies.com/v2/466b565767ca0bc444d3b8aec23be863@sha256:ddb0597e7ce5d3d1576a7260e8c0c25cca763ae257a95dee40fb3692bb4dbcc0'

30.01.19 14:56:08 (+0100) Service exited 'display sha256:c07ec8a5dfe4314a83a9e246643f21726dfc41c9b6fe0c9cd2dff0cb5a20ac08'

30.01.19 14:56:17 (+0100) Killed service 'display sha256:c07ec8a5dfe4314a83a9e246643f21726dfc41c9b6fe0c9cd2dff0cb5a20ac08'

30.01.19 14:56:18 (+0100) Deleted image 'registry.balena.catchtechnologies.com/v2/466b565767ca0bc444d3b8aec23be863@sha256:ddb0597e7ce5d3d1576a7260e8c0c25cca763ae257a95dee40fb3692bb4dbcc0'

30.01.19 14:56:18 (+0100) Deleting image 'registry.balena.catchtechnologies.com/v2/2cc24bdc82667c2b07351526eafcde9c@sha256:f3a61cde8bb3e153719a246788c575ef4b69143d5afe3b512c947d04b27c25c9'

30.01.19 14:56:22 (+0100) Deleted image 'registry.balena.catchtechnologies.com/v2/2cc24bdc82667c2b07351526eafcde9c@sha256:f3a61cde8bb3e153719a246788c575ef4b69143d5afe3b512c947d04b27c25c9'

30.01.19 14:56:23 (+0100) Installing service 'display sha256:494c109ad3567acfa22db8b08a7bb5b1116c9ccc6af15331f1ba4c5ce2a790fa'

30.01.19 14:56:34 (+0100) Installed service 'display sha256:494c109ad3567acfa22db8b08a7bb5b1116c9ccc6af15331f1ba4c5ce2a790fa'

30.01.19 14:56:34 (+0100) Starting service 'display sha256:494c109ad3567acfa22db8b08a7bb5b1116c9ccc6af15331f1ba4c5ce2a790fa'

30.01.19 14:56:42 (+0100) Started service 'display sha256:494c109ad3567acfa22db8b08a7bb5b1116c9ccc6af15331f1ba4c5ce2a790fa'

30.01.19 14:56:42 (+0100) Installing service 'catch-connect sha256:6819fd35d17517150164d43dce1fb47d60c964773b9e5098353c399c616328cc'

30.01.19 14:56:46 (+0100) Installed service 'catch-connect sha256:6819fd35d17517150164d43dce1fb47d60c964773b9e5098353c399c616328cc'

30.01.19 14:56:46 (+0100) Starting service 'catch-connect sha256:6819fd35d17517150164d43dce1fb47d60c964773b9e5098353c399c616328cc'

30.01.19 14:56:49 (+0100) Started service 'catch-connect sha256:6819fd35d17517150164d43dce1fb47d60c964773b9e5098353c399c616328cc' ```


And then it starts printing logs from my app . So the apps are downloading, just not the version we need.

patrick · January 30, 2019, 5:11pm

@richbayliss

the openbalena stack is still not considered production-ready and we cannot recommend it for a production deployment.

But the OS is production ready. The plan is to release the OS configured for our open balena instance and preloaded with an app that locks out updates.

patrick · January 30, 2019, 5:21pm

@richbayliss Thanks for the commit/release explanation.
So they should be the same, right?
The deploy and release numbers do not match for me.

Here is my current deploy output…
[Info] Compose file detected
[Info] Everything is up to date (use --build to force a rebuild)
[Info] Creating release…
[Info] Pushing images to registry…
[Info] Saving release…
[Success] Deploy succeeded!
[Success] Release: 19bc67fa2f3256944a76d92cf8de3ea5

and the app output…
$ balena app 1
== CATCH CONNECT RPI3
ID: 1
DEVICE TYPE: raspberrypi3
COMMIT: 3792751532d6788ddb6505f7f0b91d09

and the online device…
$ balena device b947ff1
== OLD FOG
ID: 31
DEVICE TYPE: raspberrypi3
STATUS: idle
IS ONLINE: true
IP ADDRESS: 192.168.1.58
APPLICATION NAME: catch-connect-rpi3
UUID: b947ff12b345db61661dc554280cbb7a
COMMIT: 3792751532d6788ddb6505f7f0b91d09
SUPERVISOR VERSION: 9.0.1
OS VERSION: balenaOS 2.29.2+rev1

patrick · January 30, 2019, 5:25pm

@richbayliss

the device will only be “pushed” an update if the VPN is connected, or once it connects back to the openbalena server to checkin (approx. every 10mins)

does the VPN need a port other than 443 to be open?

richbayliss · January 30, 2019, 5:26pm

Yes, that is correct.

Yes they should, so if yours are different then something is not working as expected when the release is made. Could you confirm that you have the latest CLI version installed; mine is 9.11.2 for reference.

richbayliss · January 30, 2019, 5:27pm

No, that should be fine for the VPN.

richbayliss · January 30, 2019, 5:30pm

Could you also try running the command like so:

balena deploy catch-connect-rpi3 --logs --source . --emulated --build

This is how I am using it, so it would be good to make sure you don’t have any issues with caching etc.

patrick · January 30, 2019, 6:30pm

$ balena version
  9.12.0

patrick · January 30, 2019, 6:33pm

$ balena deploy catch-connect-rpi3 --logs --source . --emulated --build
[Info]    Compose file detected
[Info]    Building for armv7hf/raspberrypi3
[Build]   Building services...
[Build]   catch-connect Preparing...
[Build]   display       Preparing...
[Build]   display       Step 1/6 : FROM petrosagg/resin-wpe:raspberrypi3-30c7465
[Build]   display        ---> 4b90e1c6a06a
[Build]   display       Step 2/6 : COPY udev-rules/ /etc/udev/rules.d/
[Build]   display        ---> Using cache
[Build]   display        ---> c0e6e3480b41
[Build]   display       Step 3/6 : COPY wpe-init /wpe-init
[Build]   display        ---> Using cache
[Build]   display        ---> d56a9a4392e4
[Build]   display       Step 4/6 : COPY welcome /var/lib/welcome
[Build]   display        ---> Using cache
[Build]   display        ---> 5459b667ccdd
[Build]   display       Step 5/6 : ENV WPE_URL="file:///var/lib/welcome/index.html"
[Build]   display        ---> Using cache
[Build]   display        ---> ab259a370b1a
[Build]   display       Step 6/6 : CMD [ "/wpe-init" ]
[Build]   display        ---> Using cache
[Build]   display        ---> c07ec8a5dfe4
[Build]   display       Successfully built c07ec8a5dfe4
[Build]   display       Successfully tagged catch-connect-rpi3_display:latest
[Info]    Creating release...
[Info]    Pushing images to registry...
[Info]    Saving release...
[Success] Deploy succeeded!
[Success] Release: 631712203073dc01c0ab1dc8fa17c865

$ balena app 1
== CATCH CONNECT RPI3
ID:          1
DEVICE TYPE: raspberrypi3
COMMIT:      3792751532d6788ddb6505f7f0b91d09

richbayliss · January 31, 2019, 12:59am

@patrick something else which might help us here; could you run the following command to query the API for the release you’re being told was successful…

curl -s -X GET 'https://api.balena.catchtechnologies.com/v5/release?$filter=commit%20eq%20%27631712203073dc01c0ab1dc8fa17c865%27' -H 'Authorization: Bearer <API Key>' | jq .d[0]

This will require you to have 1) generated an API key via balena api-key generate <some name> and 2) have the jq & curl utilities installed.

It would be helpful to see what comes back from that

patrick · January 31, 2019, 7:33am

richbayliss:

could you run the following command to query the API for the release you’re being told was successful…
curl -s -X GET 'https://api.balena.catchtechnologies.com/v5/release?$filter=commit%2

Screenshot also attached (a little easier on the eyes).

**{**

**"created_at"** **:** "2019-01-30T18:31:28.073Z" **,**

**"id"** **:** 24 **,**

**"belongs_to__application"** **: {**

**"__deferred"** **: {**

**"uri"** **:** "/resin/application(1)"

**},**

**"__id"** **:** 1

**},**

**"commit"** **:** "631712203073dc01c0ab1dc8fa17c865" **,**

**"composition"** **: {**

**"version"** **:** "2.1" **,**

**"volumes"** **: {**

**"catch-connect-client-data"** **: {}**

**},**

**"services"** **: {**

**"catch-connect"** **: {**

**"build"** **: {**

**"context"** **:** "./catch-connect"

**},**

**"privileged"** **:** true **,**

**"restart"** **:** "always" **,**

**"volumes"** **: [**

"catch-connect-client-data:/catch-connect-data"

**],**

**"depends_on"** **: [**

"display"

**],**

**"expose"** **: [**

"443"

**]**

**},**

**"display"** **: {**

**"build"** **: {**

**"context"** **:** "./wpe"

**},**

**"devices"** **: [**

"/dev/fb0:/dev/fb0" **,**

"/dev/vchiq:/dev/vchiq"

**],**

**"restart"** **:** "always"

**}**

**}**

**},**

**"status"** **:** "success" **,**

**"source"** **:** "local" **,**

**"start_timestamp"** **:** "2019-01-30T18:31:27.915Z" **,**

**"end_timestamp"** **:** "2019-01-30T18:31:35.387Z" **,**

**"update_timestamp"** **:** "2019-01-30T18:31:35.551Z" **,**

**"__metadata"** **: {**

**"uri"** **:** "/resin/release(24)" **,**

**"type"** **:** ""

**}**

**}**

richbayliss · January 31, 2019, 9:23am

@patrick So I have managed to replicate your experience but it has required me to change some values on my application resource via the API to do this…

If you set the flag should_track_latest_release to false via the API then whenever you deploy a new version it will not be pushed down to the device, and the commit value doesn’t update (just as-per your first post)

Could you run the following command, just to make sure the application isn’t somehow pinned to an older version:

curl -s -X GET 'https://api.balena.catchtechnologies.com/v5/application?$filter=app_name%20eq%20%27catch-connect-rpi3%27' -H 'Authorization: Bearer <API Key>' | jq .d[0]

The screenshot result is again fine. Thanks.

Topic		Replies	Views
Fail to push a new version of application to the device openBalena	2	678	August 14, 2019
Can't deploy new code openBalena	36	2428	December 3, 2019
Balena Cloud release stuck on "Factory Default" Product support raspberrypi3	2	441	October 14, 2021
balenaOS is not pulling latest release Product support	8	343	December 22, 2021
Standard flow for testing a multi-container release on a single local BalenaOS device balenaOS raspberrypi3	4	735	May 29, 2020

Latest commit not deploying

Related topics