Can't deploy new code

scarlyon · November 29, 2019, 2:56am

@shaunmulligan
So the device endpoint: curl -Ss https://api.<openbalena domain>/v5/device -H "Authorization: Bearer <api key>" | jq .

{
  "d": [
    {
      "created_at": "2019-11-22T09:23:05.171Z",
      "id": 1,
      "actor": 5,
      "uuid": "7c21238b2db06050822b91aa5e97327b",
      "local_id": null,
      "device_name": "fragrant-meadow",
      "note": null,
      "device_type": "intel-nuc",
      "belongs_to__application": {
        "__deferred": {
          "uri": "/resin/application(2)"
        },
        "__id": 2
      },
      "is_online": true,
      "last_connectivity_event": "2019-11-27T06:30:43.993Z",
      "is_connected_to_vpn": true,
      "last_vpn_event": "2019-11-27T06:30:43.993Z",
      "is_locked_until__date": null,
      "logs_channel": null,
      "public_address": null,
      "vpn_address": "10.240.0.3",
      "ip_address": "192.168.178.53",
      "is_on__commit": "1a3ef4812b54c44960e148af5cec4070",
      "download_progress": null,
      "status": "Idle",
      "os_version": "balenaOS 2.44.0+rev1",
      "os_variant": "dev",
      "supervisor_version": "10.3.7",
      "provisioning_progress": null,
      "provisioning_state": "",
      "api_port": 48484,
      "api_secret": "81d389e41e4387f154d94ff4c688cd8baec9ac541ab3f2ee3d8f4e169b0008",
      "is_managed_by__service_instance": {
        "__deferred": {
          "uri": "/resin/service_instance(5)"
        },
        "__id": 5
      },
"should_be_running__release": {
        "__deferred": {
          "uri": "/resin/release(12)"
        },
        "__id": 12
      },
      "is_managed_by__device": null,
      "__metadata": {
        "uri": "/resin/device(1)",
        "type": ""
      }
    }
  ]
}

And application endpoint: curl -Ss https://api.<openbalena domain>/v5/application -H "Authorization: Bearer <api key>" | jq .

{
  "d": [
    {
      "created_at": "2019-11-18T11:40:27.432Z",
      "id": 2,
      "actor": 4,
      "should_track_latest_release": true,
      "app_name": "testPOC",
      "slug": "testpoc",
      "device_type": "intel-nuc",
      "VPN_host": "vpn.viso.dev",
      "VPN_port": 443,
      "commit": "ddc1075a05704a9f59c85a643af0016f",
      "depends_on__application": null,
      "application_type": {
        "__deferred": {
          "uri": "/resin/application_type(0)"
        },
        "__id": 0
      },
      "__metadata": {
        "uri": "/resin/application(2)",
        "type": ""
      }
    }
  ]
}

Any idea?

shaunmulligan · November 29, 2019, 10:20am

Okay, so it doesnt look like the device or app is pinned from what I can tell. When you removed the /overlay2 folder you said it worked, was that just for the build or for the device update too? I might be that the builder or registry server on your openbalena instance is running out of space which might explain some of the odd behaviour.

scarlyon · November 29, 2019, 10:05pm

@shaunmulligan
I had to replace my local PC with a new one as the old one doesn’t have enough space.
Will try on the new one and update you.

By the way, on openBalena server:

So 100G of free space…

thgreasi · November 29, 2019, 10:27pm

That explains it, thanks for sharing your findings.
Let us know how it works on the new one.

scarlyon · December 2, 2019, 12:03am

@thgreasi @shaunmulligan
I tried on another PC, but still no luck…

Any idea?

mikesimos · December 2, 2019, 9:49am

Hi there
Did you confirm there is sufficient space on the new machine?

scarlyon · December 2, 2019, 9:55am

Of course, yes.

root@up-5:~/staged-releases# ./pin-devices-running-release.sh b7e5f80cdf18c2447aa73bbf5e53e7d3
./get-release-id.sh: line 13: jq: command not found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   113  100   113    0     0    565      0 --:--:-- --:--:-- --:--:--   567
(23) Failed writing body
setting devices with commit b7e5f80cdf18c2447aa73bbf5e53e7d3 with release =
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>SyntaxError: Unexpected token } in JSON at position 30<br> &nbsp; &nbsp;at JSON.parse (&lt;anonymous&gt;:null:null)<br> &nbsp; &nbsp;at parse (/usr/src/app/node_modules/body-parser/lib/types/json.js:89:19)<br> &nbsp; &nbsp;at /usr/src/app/node_modules/body-parser/lib/read.js:121:18<br> &nbsp; &nbsp;at invokeCallback (/usr/src/app/node_modules/raw-body/index.js:224:16)<br> &nbsp; &nbsp;at done (/usr/src/app/node_modules/raw-body/index.js:213:7)<br> &nbsp; &nbsp;at IncomingMessage.onEnd (/usr/src/app/node_modules/raw-body/index.js:273:7)<br> &nbsp; &nbsp;at IncomingMessage.emit (events.js:194:15)<br> &nbsp; &nbsp;at IncomingMessage.EventEmitter.emit (domain.js:441:20)<br> &nbsp; &nbsp;at endReadableNT (_stream_readable.js:1125:12)<br> &nbsp; &nbsp;at process._tickCallback (internal/process/next_tick.js:63:19)<br></pre>
</body>
</html>
root@up-5:~/staged-releases# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G     0  3.9G   0% /dev
tmpfs           785M  9.6M  775M   2% /run
/dev/sda2       116G   42G   69G  38% /
tmpfs           3.9G 1016K  3.9G   1% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda1       511M  3.7M  508M   1% /boot/efi
tmpfs           785M   68K  785M   1% /run/user/1000
/dev/mmcblk0p2   30G   64M   30G   1% /media/up-5/

By the way, I tried pin-devices-running-release.sh on my end, but got the error above…

Cheers,
Shane.

mikesimos · December 2, 2019, 10:19am

Hi there
I appears you 're missing some staged-releases requirements (jq) . It’s worth checking your staged-releases version is up to date, and the prerequisites are met: https://github.com/balena-io-projects/staged-releases#prerequisites .

scarlyon · December 2, 2019, 10:31am

@mikesimos
Yes, totally forgot to install it on the new PC!

Here is the current result:

root@up-5:~/staged-releases# ./pin-devices-running-release.sh b7e5f80cdf18c2447aa73bbf5e53e7d3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   113  100   113    0     0    562      0 --:--:-- --:--:-- --:--:--   565
setting devices with commit b7e5f80cdf18c2447aa73bbf5e53e7d3 with release = 23

But the device is not updated yet… (waited for 10 min… )

OKroot@up-5:~/staged-releases# balena devices
ID UUID    DEVICE NAME     DEVICE TYPE APPLICATION NAME STATUS IS ONLINE SUPERVISOR VERSION OS VERSION           DASHBOARD URL
1  7c21238 fragrant-meadow intel-nuc   testPOC          Idle   true      10.3.7             balenaOS 2.44.0+rev1 https://dashboard.viso.dev/devices/7c21238b2db06050822b91aa5e97327b/summary
root@up-5:~/staged-releases# balena device 7c21238
== FRAGRANT MEADOW
ID:                 1
DEVICE TYPE:        intel-nuc
STATUS:             inactive
IS ONLINE:          true
IP ADDRESS:         192.168.178.53
APPLICATION NAME:   testPOC
UUID:               7c21238b2db06050822b91aa5e97327b
COMMIT:             1a3ef4812b54c44960e148af5cec4070
SUPERVISOR VERSION: 10.3.7
OS VERSION:         balenaOS 2.44.0+rev1
DASHBOARD URL:      https://dashboard.viso.dev/devices/7c21238b2db06050822b91aa5e97327b/summary
root@up-5:~/staged-releases# balena device 7c21238
== FRAGRANT MEADOW
ID:                 1
DEVICE TYPE:        intel-nuc
STATUS:             inactive
IS ONLINE:          true
IP ADDRESS:         192.168.178.53
APPLICATION NAME:   testPOC
UUID:               7c21238b2db06050822b91aa5e97327b
COMMIT:             1a3ef4812b54c44960e148af5cec4070
SUPERVISOR VERSION: 10.3.7
OS VERSION:         balenaOS 2.44.0+rev1
DASHBOARD URL:      https://dashboard.viso.dev/devices/7c21238b2db06050822b91aa5e97327b/summary

A bit frustrating…

P.S. Is there anyway to build my project on the openBalena server instead of local PC?

scarlyon · December 2, 2019, 10:42am

I even tried https://github.com/balena-io-projects/simple-server-python to see how it works.
It was built without any issue, but my device is still not updated at all…

mikesimos · December 2, 2019, 10:52am

So, can you confirm you have tried the entire pipeline for starting a new project, (simple-server-python) and your setup is failing to get your app deployed? Could you provide some detailed info on the steps you 've been trying here? Thanks!

scarlyon · December 2, 2019, 10:54am

@mikesimos
I just did this:

root@up-5:/tmp# git clone https://github.com/balena-io-projects/simple-server-python
Cloning into 'simple-server-python'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 92 (delta 0), reused 3 (delta 0), pack-reused 87
Unpacking objects: 100% (92/92), done.
Checking connectivity... done.
root@up-5:/tmp# cd simple-server-python/
root@up-5:/tmp/simple-server-python# balena deploy testPOC  -b -s ./
[Info]    Creating default composition with source: /tmp/simple-server-python
[Info]    Building for amd64/intel-nuc
[Build]   Built 1 service in 0:01
[Build]   main Image size: 232.86 MB
[Info]    Creating release...
[Info]    Pushing images to registry...
[Info]    Saving release...
[Success] Deploy succeeded!
[Success] Release: f579c350bfd260c491188aa65dbf036e

                            \
                             \
                              \\
                               \\
                                >\/7
                            _.-(6'  \
                           (=___._/` \
                                )  \ |
                               /   / |
                              /    > /
                             j    < _\
                         _.-' :      ``.
                         \ r=._\        `.
                        <`\\_  \         .`-.
                         \ r-7  `-. ._  ' .  `\
                          \`,      `-.`7  7)   )
                           \/         \|  \'  / `-._
                                      ||    .'
                                       \\  (
                                        >\  >
                                    ,.-' >.'
                                   <.'_.''
                                     <'

root@up-5:/tmp/simple-server-python#

mikesimos · December 2, 2019, 11:02am

Cool! Can you confirm the testPOC app has some device(s) online?

scarlyon · December 3, 2019, 5:11am

@mikesimos

root@up-5:/tmp/simple-server-python# balena devices
ID UUID    DEVICE NAME     DEVICE TYPE APPLICATION NAME STATUS IS ONLINE SUPERVISOR VERSION OS VERSION           DASHBOARD URL
1  7c21238 fragrant-meadow intel-nuc   testPOC          Idle   true      10.3.7             balenaOS 2.44.0+rev1 https://dashboard.viso.dev/devices/7c21238b2db06050822b91aa5e97327b/summary
root@up-5:/tmp/simple-server-python# balena device 7c21238
== FRAGRANT MEADOW
ID:                 1
DEVICE TYPE:        intel-nuc
STATUS:             inactive
IS ONLINE:          true
IP ADDRESS:         192.168.178.53
APPLICATION NAME:   testPOC
UUID:               7c21238b2db06050822b91aa5e97327b
COMMIT:             1a3ef4812b54c44960e148af5cec4070
SUPERVISOR VERSION: 10.3.7
OS VERSION:         balenaOS 2.44.0+rev1
DASHBOARD URL:      https://dashboard.viso.dev/devices/7c21238b2db06050822b91aa5e97327b/summary

As you can see above, the COMMIT of 7c21238 device is not changed from 1a3ef4812b54c44960e148af5cec4070 for over a week…

I also tried the script to pin release:

root@up-5:~/staged-releases# ./pin-devices-running-release.sh f579c350bfd260c491188aa65dbf036e
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   113  100   113    0     0    549      0 --:--:-- --:--:-- --:--:--   551
setting devices with commit f579c350bfd260c491188aa65dbf036e with release = 25

But the COMMIT is not updated after an hour…

I am seeing STATUS: inactive. Does this cause this issue?

AlidaOdendaal · December 3, 2019, 11:54am

Hi

It turns out that BalenaOS 2.44.0 for intel-nuc has been pulled, the latest production release is 2.41.x. Are you able to reprovision your device with BalenaOS 2.41.1+rev1 and check if the problem still occurs?

Thanks
Alida

scarlyon · December 3, 2019, 11:17pm

@AlidaOdendaal
I have been using development images.

Phew, I was able to get it working by using node.js API: https://www.balena.io/docs/reference/sdk/node-sdk/#balena.models.device.pinToRelease

Closing this issue now…

chrisys · December 3, 2019, 11:32pm

@scarlyon thanks for letting us know